Traditional Reinforcement Learning Research Articles

Optimal tracking agent: a new framework of reinforcement learning for multiagent systems

SUMMARYThe curse of dimensionality is a ubiquitous problem for multiagent reinforcement learning, which means the learning and storing space grows exponentially with the number of agents and hinders the application of multiagent reinforcement learning. To relieve this problem, we propose a new framework named as optimal tracking agent (OTA). The OTA views the other agents as part of the environment and uses a reduced form to learn the optimal decision. Although merging other agents into the environment may reduce the dimension of action space, the environment characterized by such form is dynamic and does not satisfy the convergence of reinforcement learning (RL). Thus, we develop an estimator to track the dynamics of the environment. The estimator obtains the dynamic model, and then the model‐based RL can be used to react to the dynamic environment optimally. Because the Q‐function in OTA is also a dynamic process because of other agents’ dynamics, different from traditional RL, in which the learning is a stationary process and the usual action selection mechanisms just suit to such stationary process, we improve the greedy action selection mechanism to adapt to such dynamics. Thus, the OTA will have convergence. An experiment illustrates the validity and efficiency of the OTA.Copyright © 2012 John Wiley & Sons, Ltd.

KNN–Q(λ) algorithm-based line-grasping control of a de-icing robot

Flexible mechanical characteristics of power lines and the complex environment induce difficulties for line-grasping control of a de-icing robot when crossing obstacles. An online learning supported line-grasping control method for a de-icing robot based on the KNN– Q( λ) algorithm is presented in this paper. The proposed KNN– Q( λ) algorithm combines reinforcement learning with the k-nearest neighbor algorithm to perceive continuous states and uses a collective decision making mechanism in the action selection process to output continuous actions. Benefiting from the ability of online learning, the KNN– Q( λ)-based line-grasping control method may tolerate possible robot model errors, robot arm attitude errors and environment interferences to line-grasping control. Simulation experiment results show that as a continuous-states-continuous-actions reinforcement learning algorithm, the KNN– Q( λ) algorithm outperforms traditional reinforcement learning algorithms for the application of line-grasping control in terms of leaning rate and robustness.

Experience Replay for Real-Time Reinforcement Learning Control

Reinforcement-learning (RL) algorithms can automatically learn optimal control strategies for nonlinear, possibly stochastic systems. A promising approach for RL control is experience replay (ER), which learns quickly from a limited amount of data, by repeatedly presenting these data to an underlying RL algorithm. Despite its benefits, ER RL has been studied only sporadically in the literature, and its applications have largely been confined to simulated systems. Therefore, in this paper, we evaluate ER RL on real-time control experiments that involve a pendulum swing-up problem and the vision-based control of a goalkeeper robot. These real-time experiments are complemented by simulation studies and comparisons with traditional RL. As a preliminary, we develop a general ER framework that can be combined with essentially any incremental RL technique, and instantiate this framework for the approximate Q-learning and SARSA algorithms. The successful real-time learning results that are presented here are highly encouraging for the applicability of ER RL in practice.

Robust Quantum-Inspired Reinforcement Learning for Robot Navigation

A novel quantum-inspired reinforcement learning (QiRL) algorithm is proposed for navigation control of autonomous mobile robots. The QiRL algorithm adopts a probabilistic action selection policy and a new reinforcement strategy, which are inspired, respectively, by the collapse phenomenon in quantum measurement and amplitude amplification in quantum computation. Several simulated experiments of Markovian state transition demonstrate that QiRL is more robust to learning rates and initial states than traditional reinforcement learning. The QiRL approach is then applied to navigation control of a real mobile robot, and the simulated and experimental results show the effectiveness of the proposed approach.

Reinforcement learning via approximation of the Q-function

Relational reinforcement learning (RRL) combines traditional reinforcement learning (RL) with a strong emphasis on a relational (rather than attribute-value) representation. Earlier work used RRL on a learning version of the classic Blocks World planning problem (a version where the learner does not know what the result of taking an action will be) and the Tetris game. Learning results based on the structure of training examples were obtained, such as learning in a mixed 3–5 block environment and being able to perform in a 3 or 10 block environment. Here, we instead take a function approximation approach to RL for the Blocks World problem. We obtain similar learning accuracies, with better running times, allowing us to consider much larger problem sizes. For instance, we can train on 15 blocks and then perform well on worlds with 100–800 blocks–using less running time than the relational method required to perform well for 3–10 blocks.

Comparison of behavior-based and planning techniques on the small robot maze exploration problem

A comparison of behavior-based and planning approaches of robot control is presented in this paper. We focus on miniature mobile robotic agents with limited sensory abilities. Two reactive control mechanisms for an agent are considered-a radial basis function neural network trained by evolutionary algorithm and a traditional reinforcement learning algorithm over a finite agent state space. The control architecture based on localization and planning is compared to the former method.

Model-based Reinforcement Learning in Multiagent Systems in Extensive Forms with Perfect Information

Regarding the fact that model-based reinforcement learning has a superior performance over traditional RL, in this paper, we extend traditional model-based reinforcement learning for a group of self-interested agents with consecutive action selection trying to find the optimal policy. Every single decision making situation is modeled as extensive form games with perfect information. A modified version of prioritized sweeping is proposed in which subgame perfect equilibrium point is the optimal joint action. Finally, we discuss the algorithm analytically, and provide a formal convergence proof.

Simulation of sequential data: An enhanced reinforcement learning approach

The present study aims at contributing to the current state-of-the art of activity-based travel demand modelling by presenting a framework to simulate sequential data. To this end, the suitability of a reinforcement learning approach to reproduce sequential data is explored. Additionally, as traditional reinforcement learning techniques are not capable of learning efficiently in large state and action spaces with respect to memory and computational time requirements on the one hand, and of generalizing based on infrequent visits of all state-action pairs on the other hand, the reinforcement learning technique as used in most applications, is enhanced by means of regression tree function approximation.Three reinforcement learning algorithms are implemented to validate their applicability: the traditional Q-learning and Q-learning with bucket-brigade updating are tested against the improved reinforcement learning approach with a CART function approximator. These methods are applied on data of 26 diary days. The results are promising and show that the proposed techniques offer great opportunity of simulating sequential data. Moreover, the reinforcement learning approach improved by introducing a regression tree function approximator learns a more optimal solution much faster than the two traditional Q-learning approaches.

Quantum Reinforcement Learning

The key approaches for machine learning, particularly learning in unknown probabilistic environments, are new representations and computation mechanisms. In this paper, a novel quantum reinforcement learning (QRL) method is proposed by combining quantum theory and reinforcement learning (RL). Inspired by the state superposition principle and quantum parallelism, a framework of a value-updating algorithm is introduced. The state (action) in traditional RL is identified as the eigen state (eigen action) in QRL. The state (action) set can be represented with a quantum superposition state, and the eigen state (eigen action) can be obtained by randomly observing the simulated quantum state according to the collapse postulate of quantum measurement. The probability of the eigen action is determined by the probability amplitude, which is updated in parallel according to rewards. Some related characteristics of QRL such as convergence, optimality, and balancing between exploration and exploitation are also analyzed, which shows that this approach makes a good tradeoff between exploration and exploitation using the probability amplitude and can speedup learning through the quantum parallelism. To evaluate the performance and practicability of QRL, several simulated experiments are given, and the results demonstrate the effectiveness and superiority of the QRL algorithm for some complex problems. This paper is also an effective exploration on the application of quantum computation to artificial intelligence.

Teachable robots: Understanding human teaching behavior to build more effective robot learners

While Reinforcement Learning (RL) is not traditionally designed for interactive supervisory input from a human teacher, several works in both robot and software agents have adapted it for human input by letting a human trainer control the reward signal. In this work, we experimentally examine the assumption underlying these works, namely that the human-given reward is compatible with the traditional RL reward signal. We describe an experimental platform with a simulated RL robot and present an analysis of real-time human teaching behavior found in a study in which untrained subjects taught the robot to perform a new task. We report three main observations on how people administer feedback when teaching a Reinforcement Learning agent: (a) they use the reward channel not only for feedback, but also for future-directed guidance; (b) they have a positive bias to their feedback, possibly using the signal as a motivational channel; and (c) they change their behavior as they develop a mental model of the robotic learner. Given this, we made specific modifications to the simulated RL robot, and analyzed and evaluated its learning behavior in four follow-up experiments with human trainers. We report significant improvements on several learning measures. This work demonstrates the importance of understanding the human-teacher/robot-learner partnership in order to design algorithms that support how people want to teach and simultaneously improve the robot's learning behavior.

Case-based myopic reinforcement learning for satisfying target service level in supply chain

In the last decade, driven by global competition in the marketplace, many companies have taken initiatives to revamp their supply chains in order to increase responsiveness to changes in the marketplace. The renovation of inventory control system is central to such an effort. However, experiences in industry have shown that the control of inventory in supply chain is not an easy task because of uncertainties inherent in customer demand. In this paper, we propose a reinforcement learning algorithm appropriate for the nonstationary inventory control problem of supply chain that has a large state space. Traditional reinforcement learning algorithms such as learning automata and Q-learning have the difficulty of slow convergence when applied to the situations with large state spaces. To resolve the problems of nonstationary customer demand and large state space, we develop a case-based myopic reinforcement learning (CMRL) algorithm. A simulation-based experiment was performed to show good performance of CMRL.

Noisy K Best-Paths for Approximate Dynamic Programming with Application to Portfolio Optimization

We describe a general method to transform a non-Markovian sequential decision problem into a supervised learning problem using a K-best-paths algorithm. We consider an application in financial portfolio management where we can train a controller to directly optimize a Sharpe Ratio (or other risk-averse non-additive) utility function. We illustrate the approach by demonstrating experimental results using a kernel-based controller architecture that would not normally be considered in traditional reinforcement learning or approximate dynamic programming. We further show that using a non-additive criterion (incremental Sharpe Ratio) yields a noisy K-best-paths extraction problem, that can give substantially improved performance.

Modeling of autonomous problem solving process by dynamic construction of task models in multiple tasks environment

Traditional reinforcement learning (RL) supposes a complex but single task to be solved. When a RL agent faces a task similar to a learned one, the agent must re-learn the task from the beginning because it doesn't reuse the past learned results. This is the problem of quick action learning, which is the foundation of decision making in the real world. In this paper, we suppose agents that can solve a set of tasks similar to each other in a multiple tasks environment, where we encounter various problems one after another, and propose a technique of action learning that can quickly solve similar tasks by reusing previously learned knowledge. In our method, a model-based RL uses a task model constructed by combining primitive local predictors for predicting task and environmental dynamics. To evaluate the proposed method, we performed a computer simulation using a simple ping-pong game with variations.

Multi-robot concurrent learning of cooperative behaviours for the tracking of multiple moving targets

Reinforcement learning has been extensively studied and applied for generating cooperative behaviours in multi-robot systems. However, traditional reinforcement learning algorithms assume discrete state and action spaces with finite number of elements. This limits the learning to discrete behaviours and cannot be applied to most real multi-robot systems that inherently require appropriate combinations of different elementary behaviours. To address this problem, we design a distributed learning controller that integrates reinforcement learning with behaviour-based control networks. This learning controller can enable the robots to generate appropriate control policy without the need for human design or hardcoding. Furthermore, to address the problems in concurrent learning, we propose a distributed learning control algorithm to coordinate the concurrent learning processes. The distributed learning controller and learning control algorithm are applied to multi-robot tracking of multiple moving targets. The efficacy of our proposed scheme is shown through simulations.

Adaptive Critic Autopilot Design of Bank-to-Turn Missiles Using Fuzzy Basis Function Networks

A new adaptive critic autopilot design for bank-to-turn missiles is presented. In this paper, the architecture of adaptive critic learning scheme contains a fuzzy-basis-function-network based associative search element (ASE), which is employed to approximate nonlinear and complex functions of bank-to-turn missiles, and an adaptive critic element (ACE) generating the reinforcement signal to tune the associative search element. In the design of the adaptive critic autopilot, the control law receives signals from a fixed gain controller, an ASE and an adaptive robust element, which can eliminate approximation errors and disturbances. Traditional adaptive critic reinforcement learning is the problem faced by an agent that must learn behavior through trial-and-error interactions with a dynamic environment, however, the proposed tuning algorithm can significantly shorten the learning time by online tuning all parameters of fuzzy basis functions and weights of ASE and ACE. Moreover, the weight updating law derived from the Lyapunov stability theory is capable of guaranteeing both tracking performance and stability. Computer simulation results confirm the effectiveness of the proposed adaptive critic autopilot.

Reinforcement learning algorithms for robotic navigation in dynamic environments

The purpose of this study was to examine improvements to reinforcement learning (RL) algorithms in order to successfully interact within dynamic environments. The scope of the research was that of RL algorithms as applied to robotic navigation. Proposed improvements include: addition of a forgetting mechanism, use of feature based state inputs, and hierarchical structuring of an RL agent. Simulations were performed to evaluate the individual merits and flaws of each proposal, to compare proposed methods to prior established methods, and to compare proposed methods to theoretically optimal solutions. Incorporation of a forgetting mechanism did considerably improve the learning times of RL agents in a dynamic environment. However, direct implementation of a feature-based RL agent did not result in any performance enhancements, as pure feature-based navigation results in a lack of positional awareness, and the inability of the agent to determine the location of the goal state. Inclusion of a hierarchical structure in an RL agent resulted in significantly improved performance, specifically when one layer of the hierarchy included a feature-based agent for obstacle avoidance, and a standard RL agent for global navigation. In summary, the inclusion of a forgetting mechanism, and the use of a hierarchically structured RL agent offer substantially increased performance when compared to traditional RL agents navigating in a dynamic environment.

An Use of Reinforcement Learning in a Multi-Agent Environment.

In order to give more adaptability to a multiagent system, it would be desirable for each agent to have some kind of online learning ability for how to help and guide the other agents, i.e., how to cooperate with each other for the purpose of achieving a task which is given to the whole system. When we browse the field of computational methods of online learning, reinforcement learning seems a good candidate for this mechanism. However, due to the theoretical limitation that it assumes that an environment is Markovian, traditional reinforcement learning algorithms cannot be applied directly to this behavior acquisition problem in a multiagent environment. In this paper, an online learning mechanism is designed by using two reinforcement learning units. The first one is for predicting the move of the other agents at the next time-step, and the second one is for building the appropriate action rule set for an agent itself. Several computer simulations of the cooperative carrying problem are conducted to investigate the effectiveness of the proposed approach.

A reinforcement learning adaptive fuzzy controller for robots

In this paper, a new reinforcement learning scheme is developed for a class of serial-link robot arms. Traditional reinforcement learning is the problem faced by an agent that must learn behavior through trial-and-error interactions with a dynamic environment. In the proposed reinforcement learning scheme, an agent is employed to collect signals from a fixed gain controller, an adaptive critic element and a fuzzy action-generating element. The action generating element is a fuzzy approximator with a set of tunable parameters, and the performance measurement mechanism sends an error metric to the adaptive critic element for generating and transferring a reinforcement learning signal to the agent. Moreover, a tuning algorithm of the proposed scheme that can guarantee both tracking performance and stability is derived from the Lyapunov stability theory. Therefore, a combination of adaptive fuzzy control and reinforcement learning scheme is also concerned with algorithms for eliminating a sequence of decisions from experience. Simulations of the proposed reinforcement adaptive fuzzy control scheme on the cart-pole balancing problem and a two-degree-of freedom (2DOF) manipulator, SCARA robot arm verify the effectiveness of our approach.

Robot learning with GA-based fuzzy reinforcement learning agents

Abstract How to learn from both expert knowledge and measurement-based information for a robot to acquire perception and motor skills is a challenging research topic in the field of autonomous robotic systems. For this reason, a general GA (genetic algorithm)-based fuzzy reinforcement learning (GAFRL) agent is proposed in this paper. We first characterize the robot learning problem and point out some major issues that need to be addressed in conjunction with reinforcement learning. Based on a neural fuzzy network architecture of the GAFRL agent, we then discuss how different kinds of expert knowledge and measurement-based information can be incorporated in the GAFRL agent so as to accelerate its learning. By making use of the global optimization capability of GAs, the GAFRL can solve the local minima problem in traditional actor-critic reinforcement learning. On the other hand, with the prediction capability of the critic network, GAs can evaluate the candidate solutions regularly even during the periods without external feedback from the environment. This can guide GAs to perform a more effective global search. Finally, different types of GAFRL agents are constructed and verified using the simulation model of a physical biped robot.

Incorporation of perception-based information in robot learning using fuzzy reinforcement learning agents

Robot learning in unstructured environments has been proved to be an extremely challenging problem, mainly because of many uncertainties always present in the real world. Human beings, on the other hand, seem to cope very well with uncertain and unpredictable environments, often relying on perception-based information. Furthermore, humans beings can also utilize perceptions to guide their learning on those parts of the perception-action space that are actually relevant to the task. Therefore, we conduct a research aimed at improving robot learning through the incorporation of both perception-based and measurement-based information. For this reason, a fuzzy reinforcement learning (FRL) agent is proposed in this paper. Based on a neural-fuzzy architecture, different kinds of information can be incorporated into the FRL agent to initialise its action network, critic network and evaluation feedback module so as to accelerate its learning. By making use of the global optimisation capability of GAs (genetic algorithms), a GA-based FRL (GAFRL) agent is presented to solve the local minima problem in traditional actor-critic reinforcement learning. On the other hand, with the prediction capability of the critic network, GAs can perform a more effective global search. Different GAFRL agents are constructed and verified by using the simulation model of a physical biped robot. The simulation analysis shows that the biped learning rate for dynamic balance can be improved by incorporating perception-based information on biped balancing and walking evaluation. The biped robot can find its application in ocean exploration, detection or sea rescue activity, as well as military maritime activity.