Actor-critic Learning Research Articles

INFLUENCE OF TEMPERATURE ON SWARMBOTS THAT LEARN

The problem considered in this article is how a cybernetic system can learn to control its actions in a hostile environment. This article focuses on an approach to solving this problem in an environment with varying temperatures. In effect, machines that operate outdoors have higher survivability if actions are chosen during periods when it is cooler (e.g., night-time or early morning rather than mid- to late afternoon during summer months). The assumption made here is that learning to choose actions that compensate for the influence of temperature has beneficial influence on the functioning of individuals in robot societies (collections of cooperating robots called swarmbots or swarms). In keeping with this idea, a biologically-inspired form of adaptive learning is given in this article. Conventional actor-critic learning provides a framework for the control strategy introduced in this article. It is ethology (study of behavior of organisms) that provides a basis for monitoring the behavior of a swarmbot. Individual behaviours together with sensor measurements are recorded in tables called ethograms. Swarm behavior tends to be episodic. An ethogram is recorded during each episode during the lifespan of a swarm. Each ethogram is a source of measurements that can be used to influence learning during an episode. The contribution of this article is the introduction of a biologically-inspired approach to learning that adapts to changing temperatures.

The Actor-Critic Learning Is Behind the Matching Law: Matching Versus Optimal Behaviors

The ability to make a correct choice of behavior from various options is crucial for animals' survival. The neural basis for the choice of behavior has been attracting growing attention in research on biological and artificial neural systems. Alternative choice tasks with variable ratio (VR) and variable interval (VI) schedules of reinforcement have often been employed in studying decision making by animals and humans. In the VR schedule task, alternative choices are reinforced with different probabilities, and subjects learn to select the behavioral response rewarded more frequently. In the VI schedule task, alternative choices are reinforced at different average intervals independent of the choice frequencies, and the choice behavior follows the so-called matching law. The two policies appear robustly in subjects' choice of behavior, but the underlying neural mechanisms remain unknown. Here, we show that these seemingly different policies can appear from a common computational algorithm known as actor-critic learning. We present experimentally testable variations of the VI schedule in which the matching behavior gives only a suboptimal solution to decision making and show that the actor-critic system exhibits the matching behavior in the steady state of the learning even when the matching behavior is suboptimal. However, it is found that the matching behavior can earn approximately the same reward as the optimal one in many practical situations.

Passive dynamic walker controller design employing an RLS-based natural actor–critic learning algorithm

A passive dynamic walker belongs to a class of bipedal walking robots that are able to walk stably down a small decline without using any actuators. The purpose of this research is to design a controller in order to build actuated robots capable of walking on a flat terrain based on passive dynamic walking. To achieve this objective, a control algorithm was used based on reinforcement learning (RL). The RL method is a goal-directed learning of a mapping from situations to actions without relying on exemplary supervision or complete models of the environment. The goal of the RL method is to maximize a reward, which is an evaluative feedback from the environment. In the process of constructing the reward of the actuated passive dynamic walker, the control objective, which is stable walking on level ground, is directly included. In this study, an RL algorithm based on the actor–critic architecture and the natural gradient method is applied. Also, the recursive least-squares (RLS) method was employed for the learning process in order to improve the efficiency of the method. The control algorithm was verified with computer simulations based on the eigenvalue analysis for stable locomotion.

A Proposal of Adaptive PID Controller Based on Reinforcement Learning

Abstract Aimed at the lack of self-tuning PID parameters in conventional PID controllers, the structure and learning algorithm of an adaptive PID controller based on reinforcement learning were proposed. Actor-Critic learning was used to tune PID parameters in an adaptive way by taking advantage of the model-free and on-line learning properties of re-inforcement learning effectively. In order to reduce the demand of storage space and to improve the learning efficiency, a single RBF neural network was used to approximate the policy function of Actor and the value function of Critic si-multaneously. The inputs of RBF network are the system error, as well as the first and the second-order differences of error. The Actor can realize the mapping from the system state to PID parameters, while the Critic evaluates the outputs of the Actor and produces TD error. Based on TD error performance index and gradient descent method, the updating rules of RBF kernel function and network weights were given. Simulation results show that the proposed controller is efficient for complex nonlinear systems and it is perfectly adaptable and strongly robust, which is better than that of a conventional PID controller.

Obstacle Avoidance of a Mobile Robot Using Hybrid Learning Approach

in this paper, a hybrid learning approach for obstacle avoidance of a mobile robot is presented. the key features of the approach are, firstly, innate hardwired behaviors which are used to bootstrap learning in the mobile robot system. a neuro-fuzzy controller is developed from a pre-wired or innate controller based on supervised learning in a simulation environment. the fuzzy inference system has been constructed based on the generalized dynamic fuzzy neural networks learning algorithm of Wu and Er, whereby structure and parameters identification are carried out automatically and simultaneously. Secondly, the neuro-fuzzy controller is capable of re-adapting in a new environment. After carrying out the learning phase on a simulated robot, the controller is implemented on a real robot. A reinforcement learning method based on the fuzzy actor-critic learning algorithm is employed so that the system can re-adapt to a new environment without human intervention. In this phase, the structure of the fuzzy inference system and the parameters of the antecedent parts of fuzzy rules are frozen, and reinforcement learning is applied to further tune the parameters in the consequent parts of the fuzzy rules. Through the hybrid learning approach, an efficient and compact neuro-fuzzy system is generated for obstacle avoidance of a mobile robot in the real world.

Adaptive actor-critic learning for the control of mobile robots by applying predictive models

In this paper, we propose two methods of adaptive actor-critic architectures to solve control problems of nonlinear systems. One method uses two actual states at time k and time k+1 to update the learning algorithm. The basic idea of this method is that the agent can directly take some knowledge from the environment to improve its knowledge. The other method only uses the state at time k to update the algorithm. This method is called, learning from prediction (or simulated experience). Both methods include one or two predictive models, which are assumed to be applied to construct predictive states and a model-based actor (MBA). Here, the MBA as an actor can be viewed as a network where the connection weights are the elements of the feedback gain matrix. In the critic part, two value-functions are realized as a pure static mapping, which can be reduced to a nonlinear current estimator by using the radial basis function neural networks (RBFNNs). Simulation results obtained for a dynamical model of nonholonomic mobile robots with two independent driving wheels are presented. They show the effectiveness of the proposed approaches for the trajectory tracking control problem.

Application of actor-critic learning algorithm for optimal bidding problem of a genco

The optimal bidding for generation companies (GenCo) in the deregulated power market is an involved task. The problem is formulated in the framework of the Markov decision process (MDP), a discrete stochastic optimization method. When the time span considered is 24 hours, the temporal difference method becomes attractive for application. The cumulative profit over the span is the objective function to be optimized. The temporal difference technique and actor-critic learning algorithm is employed. An optimal strategy is devised to maximize the profit. A market-clearing system is included in the formulation. Simulation cases of three, seven, and ten participants are considered, and the results obtained are discussed.

Application of Actor-Critic Learning Algorithm for Optimal Bidding Problem of a Genco

The optimal bidding for generation companies (GenCo) in the deregulated power market is an involved task. The problem is formulated in the framework of the Markov decision process (MDP), a discrete stochastic optimization method. When the time span considered is 24 hours, the temporal difference method becomes attractive for application. The cumulative profit over the span is the objective function to be optimized. The temporal difference technique and actor-critic learning algorithm is employed. An optimal strategy is devised to maximize the profit. A market-clearing system is included in the formulation. Simulation cases of three, seven, and ten participants are considered, and the results obtained are discussed.

Structure in the Space of Value Functions

Solving in an efficient manner many different optimal control tasks within the same underlying environment requires decomposing the environment into its computationally elemental fragments. We suggest how to find fragmentations using unsupervised, mixture model, learning methods on data derived from optimal value functions for multiple tasks, and show that these fragmentations are in accord with observable structure in the environments. Further, we present evidence that such fragments can be of use in a practical reinforcement learning context, by facilitating online, actor-critic learning of multiple goals MDPs.

2P1-L8 actor-criticによる複数エージェントの協調行動学習(27. 強化学習とロボティクス・メカトロニクス)

2P1-L8 actor-criticによる複数エージェントの協調行動学習(27. 強化学習とロボティクス・メカトロニクス)

Building up Embodiment in Learning Agents Using A Gaussian Radial Basis Function Neural Network

In this paper, we propose actor-critic learning with adaptive state space construction. The gaussian radial basis function neural network is employed for both the actor and the critic modules, where each hidden neuron covers a subspace of the sensor space and so the hidden layer corresponds to the state space. In the proposed algorithm, a robot starts without any states and a new state is generated incrementally by adding a new hidden neuron. One clear advantage of the proposed algorithm to others is the performance improvement by the minimal training after adding a new state, i.e. the adjustment of the connective strength between the new neuron and others is only required after adding a new hidden neuron. This provides an efficient method to construct the state space during the learning in the real world. Resulting state space represents aspects of the environment in which the robot works and the characteristic of the robot itself. In this sense, the obtained state space is said to represent the embodiment of the robot.

Fuzzy inference system learning by reinforcement methods

Fuzzy Actor-Critic Learning (FACL) and Fuzzy Q-Learning (FQL) are reinforcement learning methods based on dynamic programming (DP) principles. In the paper, they are used to tune online the conclusion part of fuzzy inference systems (FIS). The only information available for learning is the system feedback, which describes in terms of reward and punishment the task the fuzzy agent has to realize. At each time step, the agent receives a reinforcement signal according to the last action it has performed in the previous state. The problem involves optimizing not only the direct reinforcement, but also the total amount of reinforcements the agent can receive in the future. To illustrate the use of these two learning methods, we first applied them to a problem that involves finding a fuzzy controller to drive a boat from one bank to another, across a river with a strong nonlinear current. Then, we used the well known Cart-Pole Balancing and Mountain-Car problems to be able to compare our methods to other reinforcement learning methods and focus on important characteristic aspects of FACL and FQL. We found that the genericity of our methods allows us to learn every kind of reinforcement learning problem (continuous states, discrete/continuous actions, various type of reinforcement functions). The experimental studies also show the superiority of these methods with respect to the other related methods we can find in the literature.

Incremental multi-step Q-learning

This paper presents a novel incremental algorithm that combines Q-learning, a well-known dynamic-programming based reinforcement learning method, with the TD(λ) return estimation process, which is typically used in actor-critic learning, another well-known dynamic-programming based reinforcement learning method. The parameter λ is used to distribute credit throughout sequences of actions, leading to faster learning and also helping to alleviate the non-Markovian effect of coarse state-space quatization. The resulting algorithm.Q(λ)-learning, thus combines some of the best features of the Q-learning and actor-critic learning paradigms. The behavior of this algorithm has been demonstrated through computer simulations.