Conventional Reinforcement Learning Methods Research Articles

Water management scheme based on prioritized deep deterministic policy gradient for proton exchange membrane fuel cells

AbstractTo effectively tackle the intricate and dynamic challenges encountered in proton exchange membrane fuel cells (PEMFCs), this paper introduces a model-free reinforcement learning approach to address its water management issue. Recognizing the limitations of conventional reinforcement learning methods such as Q-learning in handling the continuous actions and nonlinearity inherent in PEMFCs water management, we propose a prioritized deep deterministic policy gradient (DDPG) method. This method, rooted in the Actor-Critic framework, leverages double neural networks and prioritized experience replay to enable adaptive water management and balance. Additionally, we establish a PEMFCs water management platform and implement the prioritized DDPG method using "Tianshou", a modularized Python library for deep reinforcement learning. Through experimentation, the effectiveness of our proposed method is verified. This study contributes to advancing the understanding and management of water dynamics in PEMFCs, offering a promising avenue for enhancing their performance and reliability.

An efficient reinforcement learning scheme for the confinement escape problem

Crucial real-world problems in robotics like trajectory planning during convoy missions and autonomous rescue missions can be framed as a confinement escape problem (CEP) (a type of pursuit-evasion game). In a typical CEP, an evader attempts to escape a confinement region by sequentially making decisions to plan an escape while the region is patrolled by multiple smart pursuers. The evader has a limited sensing range and does not know the total number of pursuers and their pursuit strategies making it difficult to model the environment and obtain a generalizable escape strategy. In this paper, the CEP is formulated in a reinforcement learning (RL) framework to overcome the above difficulties. The state function is designed independent of the total number of pursuers and the shape of the confinement region thereby making the RL approach scalable. To handle training consistency issues in deep RL and convergence issues due to sparse rewards, a Scaffolding Reflection based Reinforcement Learning (SR2L) approach is presented in this paper where the SR2L employs an actor–critic method with a motion planner scaffold to accelerate its training speed. Performance evaluation of SR2L shows that it trains twice as fast compared to other existing state-of-the-art actor–critic RL methods. Performance results show that the convergence of SR2L is more consistent than the corresponding conventional actor–critic RL methods and interactive reinforcement learning methods. Monte-Carlo simulation results show that SR2L outperforms other conventional RL methods and the motion planner with at least 28% and 10% faster escape times respectively while having the lowest variance in escape times against different pursuit strategies. Ablation studies done by changing different environmental parameters clearly show the scalability and generalizability of the proposed SR2L approach.

Multi-step motion learning by combining learning-from-demonstration and policy-search

ABSTRACT In this paper, we focus on tasks that require multi-step motions to achieve the goal (defined as a ‘multi-step task’), and we describe a method for a robot to automatically achieve the final goal of a multi-step task. We proposed a method based on reinforcement learning and ‘Teaching by Showing’ for multi-step tasks. A robot can learn how to complete a task automatically by referring to the motions of a human operator, even if the task consists of multi-step motions. Because a human operator is not required to operate the robot during the learning process, we believe that our proposed method can reduce the burden on the human operator. Finally, we conducted experiments to validate the effectiveness of the proposed method and compared it to a conventional reinforcement learning method.

Self-Supervised Reinforcement Learning for Active Object Detection

Active object detection (AOD) offers significant advantage in expanding the perceptual capacity of a robotics system. AOD is formulated as a sequential action decision process to determine optimal viewpoints to identify objects of interest in a visual scene. While reinforcement learning (RL) has been successfully used to solve many AOD problems, conventional RL methods suffer from (i) sample inefficiency, and (ii) unstable outcome due to inter-dependencies of action type (direction of view change) and action range (step size of view change). To address these issues, we propose a novel self-supervised RL method, which employs self-supervised representations of viewpoints to initialize the policy network, and a self-supervised loss on action range to enhance the network parameter optimization. The output and target pairs of self-supervised learning loss are automatically generated from the policy network online prediction and a range shrinkage algorithm (RSA), respectively. The proposed method is evaluated and benchmarked on two public datasets (T-LESS and AVD) using on-policy and off-policy RL algorithms. The results show that our method enhances detection accuracy and achieves faster convergence on both datasets. By evaluating on a more complex environment with a larger state space (where viewpoints are more densely sampled), our method achieves more robust and stable performance. Our experiment on real robot application scenario to disambiguate similar objects in a cluttered scene has also demonstrated the effectiveness of the proposed method.

Reward-Reinforced Generative Adversarial Networks for Multi-Agent Systems

Multi-agent systems deliver highly resilient and adaptable solutions for common problems in telecommunications, aerospace, and industrial robotics. However, achieving an optimal global goal remains a persistent obstacle for collaborative multi-agent systems, where learning affects the behaviour of more than one agent. A number of nonlinear function approximation methods have been proposed for solving the Bellman equation, which describe a recursive format of an optimal policy. However, how to leverage the value distribution based on reinforcement learning, and how to improve the efficiency and efficacy of such systems remain a challenge. In this work, we developed a reward-reinforced generative adversarial network to represent the distribution of the value function, replacing the approximation of Bellman updates. We demonstrated our method is resilient and outperforms other conventional reinforcement learning methods. This method is also applied to a practical case study: maximising the number of user connections to autonomous airborne base stations in a mobile communication network. Our method maximises the data likelihood using a cost function under which agents have optimal learned behaviours. This reward-reinforced generative adversarial network can be used as a generic framework for multi-agent learning at the system level.

Hierarchical Reinforcement Learning for Relay Selection and Power Optimization in Two-Hop Cooperative Relay Network

In this paper, we study the outage probability minimizing problem in a two-hop cooperative relay network. To reduce outage probability, existing studies propose many schemes for relay selection and power allocation, which are usually based on the assumption of exact channel state information (CSI). However, it is difficult to obtain perfect instantaneous CSI in practical situations where channel states change rapidly, and thus traditional methods would not perform well. Considering these factors, we turn to the emerging reinforcement learning (RL) methods for solutions. RL methods do not need any prior knowledge of CSI, but use neural network for approximation and decision after interacting with communication environment. Nevertheless, conventional RL methods, including most deep reinforcement learning (DRL) methods, cannot perform well when the search space is too large. In addition, non-stationarity is a common problem when using hierarchical reinforcement learning (HRL), which is caused by the changing behavior in different hierarchies. Therefore, we first propose a DRL framework with an outage-based reward function, which is then used as a baseline. Then, we further design an HRL framework and training algorithm. By decomposing relay selection and power allocation into two hierarchical optimization objectives, and combining on- policy and off-policy methods in the HRL framework, our method successfully address the sparse reward and non-stationary problem. Simulation results reveal that compared with traditional DRL method, the proposed HRL training algorithm can converge faster and reduce the outage probability by 8% in two-hop relay network with the same outage threshold.

Active Inference Integrated With Imitation Learning for Autonomous Driving

Classical imitation learning methods suffer substantially from the learning hierarchical policies when the imitative agent faces an unobserved state by the expert agent. To address these drawbacks, we propose an online active learning through active inference approach that encodes the expert’s demonstrations based on observation-action to improve the learner’s future motion prediction. For this purpose, we provide a switching Dynamic Bayesian Network based on the dynamic interaction between the expert agent and another object in its surrounding as a reference model, which we exploit to initialize an incremental probabilistic learning model. This learning model grows and matures based on the free-energy formulation and message passing of active inference dynamically at discrete and continuous levels in an online active learning phase. In this scheme, generalized states of the learning world are represented as distance-vector, where it is the learner’s observation concerning its interaction with a moving object. Considering the distance vector entail intentions, it enables action prediction evaluation in a prospective sense. We illustrate these points using simulations of driving intelligent agents. The learning agent is trained by using long-term predictions from the generative learning model to reproduce the expert’s motion while learning how to select a suitable action through new experiences. Our results affirm that a Dynamic Bayesian optimal approach provides a principled framework and outperforms conventional reinforcement learning methods. Furthermore, it endorses the general formulation of action prediction as active inference.

Reinforcement Learning Based Fast Self-Recalibrating Decoder for Intracortical Brain-Machine Interface.

Background: For the nonstationarity of neural recordings in intracortical brain–machine interfaces, daily retraining in a supervised manner is always required to maintain the performance of the decoder. This problem can be improved by using a reinforcement learning (RL) based self-recalibrating decoder. However, quickly exploring new knowledge while maintaining a good performance remains a challenge in RL-based decoders. Methods: To solve this problem, we proposed an attention-gated RL-based algorithm combining transfer learning, mini-batch, and weight updating schemes to accelerate the weight updating and avoid over-fitting. The proposed algorithm was tested on intracortical neural data recorded from two monkeys to decode their reaching positions and grasping gestures. Results: The decoding results showed that our proposed algorithm achieved an approximate 20% increase in classification accuracy compared to that obtained by the non-retrained classifier and even achieved better classification accuracy than the daily retraining classifier. Moreover, compared with a conventional RL method, our algorithm improved the accuracy by approximately 10% and the online weight updating speed by approximately 70 times. Conclusions: This paper proposed a self-recalibrating decoder which achieved a good and robust decoding performance with fast weight updating and might facilitate its application in wearable device and clinical practice.

Research on parameter optimisation of dynamic priority scheduling algorithm based on improved reinforcement learning

The dynamic priority scheduling algorithm is one of the real-time scheduling algorithms in a power system. However, it ignores the impact of the weight of each index when selecting indicators that affect scheduling performance. There is no definite objective function relation between weight parameters and scheduling performance. Hence, the heuristic algorithm is difficult to optimise the weight parameters. Aiming to solve this problem, a dynamic priority scheduling algorithm based on improved reinforcement learning (RL) is proposed for parameter optimisation. By learning from each other, the weighting parameters and the deadline miss rate (DMR), the global optimisation of weighting parameters can be achieved, but the learning efficiency of the conventional RL method is low. According to the task scheduling performance (the DMR) and the task characteristics, this study improves the RL action step and reward function, which accelerates the online learning speed and improves the optimisation ability of the RL algorithm. Experimental results show that the improved RL algorithm not only optimises the weight parameters but reduces the DMR, which reduces the number of iterations of RL. A scheduling algorithm optimised by RL can be better applied to industrial control and power system resource scheduling, which not only improves control efficiency but reduces scheduling costs.

Heurisztikusan gyorsított megerősítéses tanulási módszerek - áttekintés

The conventional reinforcement learning methods (e.g. Q-learning, SARSA) search the solution through trial and error by the properly defined reward function, therefore based on reinforcements given by the environment. The beginning of the learning phase the system does not have any knowledge about the solution, the goal of these methods are to build the knowledgebase during the learning phase. Thus the learning phase can be a long task and the number of iterations which lead to the final solution can be high. In that case the learning process can be speed up, if there are portion of knowledge about the solution and it can be injected into the learning system. The heuristically accelerated reinforcement learning methods incorporate the knowledge defined by human, due to this reason the convergence speed of the system and the number of iterations can be decreased. The main goal of the paper is to give an overview about reinforcement learning methods which heuristically accelerated and give a possibility to inject human defined knowledge into the learning system.

An Experience Aggregative Reinforcement Learning With Multi-Attribute Decision-Making for Obstacle Avoidance of Wheeled Mobile Robot

A variety of reinforcement learning (RL) methods are developed to achieve the motion control for the robotic systems, which has been a hot issue. However, the performance of the conventional RL methods often encounters a bottleneck, because the robots have difficulty in choosing an appropriate action in the control task due to the exploration-exploitation dilemma. To address this problem and improve the learning performance, this work introduces an experience aggregative reinforcement learning method with a Multi-Attribute Decision-Making (MADM) to achieve the real-time obstacle avoidance of wheeled mobile robot (WMR). The proposed method employs an experience aggregation method to cluster experiential samples and it can achieve more effective experience storage. Moreover, to achieve the effective action selection using the prior experience, an action selection policy based on a Multi-Attribute Decision-Making is proposed. Inspired by the hierarchical decision-making, this work decomposes the original obstacle avoidance task into two sub-tasks using a divide-and-conquer approach. Each sub-task is trained individually by a double Q-learning using a simple reward function. Each sub-task learns an action policy, which enables the sub-task to selects an appropriate action to achieve a single goal. The standardized rewards of sub-tasks are calculated when fusing these sub-tasks to eliminate differences in rewards for sub-tasks. Then, the proposed method integrates the prior experience of three trained sub-tasks via an action policy based on a MADM to complete the source task. Simulation results show that the proposed method outperforms competitors.

Deep Reinforcement Learning for Strategic Bidding in Electricity Markets

Bi-level optimization and reinforcement learning (RL) constitute the state-of-the-art frameworks for modeling strategic bidding decisions in deregulated electricity markets. However, the former neglects the market participants’ physical non-convex operating characteristics, while conventional RL methods require discretization of state and/or action spaces and thus suffer from the curse of dimensionality. This paper proposes a novel deep reinforcement learning (DRL) based methodology, combining a deep deterministic policy gradient (DDPG) method with a prioritized experience replay (PER) strategy. This approach sets up the problem in multi-dimensional continuous state and action spaces, enabling market participants to receive accurate feedback regarding the impact of their bidding decisions on the market clearing outcome, and devise more profitable bidding decisions by exploiting the entire action domain, also accounting for the effect of non-convex operating characteristics. Case studies demonstrate that the proposed methodology achieves a significantly higher profit than the alternative state-of-the-art methods, and exhibits a more favourable computational performance than benchmark RL methods due to the employment of the PER strategy.

Analysis of Agent Expertise in Ms. Pac-Man Using Value-of-Information-Based Policies

Conventional reinforcement learning methods for Markov decision processes rely on weakly-guided, stochastic searches to drive the learning process. It can therefore be difficult to predict what agent behaviors might emerge. In this paper, we consider an information-theoretic cost function for performing constrained stochastic searches that promote the formation of risk-averse to risk-favoring behaviors. This cost function is the value of information, which provides the optimal trade-off between the expected return of a policy and the policy's complexity; policy complexity is measured by number of bits and controlled by a single hyperparameter on the cost function. As the policy complexity is reduced, the agents will increasingly eschew risky actions. This reduces the potential for high accrued rewards. As the policy complexity increases, the agents will take actions, regardless of the risk, that can raise the long-term rewards. The obtainable reward depends on a single, tunable hyperparameter that regulates the degree of policy complexity. We evaluate the performance of value-of-information-based policies on a stochastic version of Ms. Pac-Man. A major component of this paper is the demonstration that ranges of policy complexity values yield different game-play styles and explaining why this occurs. We also show that our reinforcement-learning search mechanism is more efficient than the others we utilize. This result implies that the value of information theory is appropriate for framing the exploitation-exploration trade-off in reinforcement learning.

Multi-Agent Deep Reinforcement Learning-Based Cooperative Spectrum Sensing With Upper Confidence Bound Exploration

In this paper, a multi-agent deep reinforcement learning method was adopted to realize cooperative spectrum sensing in cognitive radio networks. Each secondary user learns an efficient sensing strategy from the sensing results of some of the selected spectra to avoid interference to the primary users and to coordinate with other secondary users. It is necessary to balance exploration and exploitation in the learning process when using deep reinforcement learning methods, helping explain that upper confidence bound with Hoeffding-style bonus has been adopted in this paper to improve the efficiency of exploration. The simulation results verify that the proposed algorithm, when compared with the conventional reinforcement learning methods with <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\varepsilon $ </tex-math></inline-formula> -greedy exploration, is much easier to achieve faster convergence speed and better reward performance.

Reinforcement learning for dynamic environment: a classification of dynamic environments and a detection method of environmental changes

Engineers and researchers are paying more attention to reinforcement learning (RL) as a key technique for realizing computational intelligence such as adaptive and autonomous decentralized systems. In general, it is not easy to put RL into practical use. In prior research our approach mainly dealt with the problem of designing state and action spaces and we have proposed an adaptive co-construction method of state and action spaces. However, it is more difficult to design state and action spaces in dynamic environments than in static ones. Therefore, it is even more effective to use an adaptive co-construction method of state and action spaces in dynamic environments. In this paper, our approach mainly deals with a problem of adaptation in dynamic environments. First, we classify tasks of dynamic environments and propose a detection method of environmental changes to adapt to dynamic environments. Next, we conducted computational experiments using a so-called "path planning problem" with a slowly changing environment where the aging of the system is assumed. The performances of a conventional RL method and the proposed detection method were confirmed.

Incremental Sparse Bayesian Method for Online Dialog Strategy Learning

This paper proposes an incremental sparse Bayesian learning method to allow continuous dialog strategy learning from the interactions with real users. Since conventional reinforcement learning (RL) methods require a huge number of dialogs to reach convergence, it has been essential to use a simulated user in training dialog policies. The disadvantage of this approach is that the trained dialog policies always lag behind the optimal one for live users. In order to tackle this problem, a few studies applying online RL methods to dialog management have emerged and showed very promising results. However, these methods are limited to learning online the weight parameters of the basis functions in the model and so need batch learning on a fixed data set or some heuristics to find appropriate values for other meta parameters such as sparsity-controlling thresholds, basis function parameters, and noise parameters. The proposed method attempts to overcome this limitation to achieve fully incremental and fast dialog strategy learning by adopting a sparse Bayesian learning method for value function approximation. In order to verify the proposed method, three different experimental conditions have been used: artificial data, a simulated user, and real users. The experiment on the artificial data showed that the proposed method successfully learns all the parameters in an incremental manner. Also, the experiment on training and evaluating dialog policies with a simulated user clearly demonstrated that the proposed method is much faster than conventional RL methods. A live user study showed that the dialog strategy learned from real users performed as good as the best past systems, although it slightly underperformed the one trained on simulated dialogs due to the difficulty of user feedback elicitation.

A state predictor‐based reinforcement learning system

AbstractThe present paper proposes a new reinforcement learning (RL) system called a state predictor‐based RL system in order to solve the explosion of state space and create cooperative behaviors in multiagent systems. The proposed system realizes a predictive function by representing both the present and the next state‐action groups with ITPM, an incremental topology map. The proposed system is applied to the pursuit problem, and its performance is evaluated by comparison with the conventional RL method in computer simulations. The experimental results show that the proposed system can appropriately learn in a complex environment which is hardly solved by the conventional RL. Furthermore, it is confirmed that the proposed system can acquire cooperative strategies. © 2010 Wiley Periodicals, Inc. Electron Comm Jpn, 93(6): 8–18, 2010; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/ecj.10258

状態予測型強化学習システム

The present paper proposes a new reinforcement learning (RL) system called a state predictor-based RL system in order to solve the explosion of state space and create cooperative behaviors in multiagent systems. The proposed system realizes a predictive function by representing both the present and the next state-action groups with ITPM, an incremental topology map. The proposed system is applied to the pursuit problem, and its performance is evaluated by comparison with the conventional RL method in computer simulations. The experimental results show that the proposed system can appropriately learn in a complex environment which is hardly solved by the conventional RL. Furthermore, it is confirmed that the proposed system can acquire cooperative strategies. © 2010 Wiley Periodicals, Inc. Electron Comm Jpn, 93(6): 8–18, 2010; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/ecj.10258

2A1-M09 学習に基づく3自由度1脚跳躍ロボットの運動制御に関する研究(進化・学習とロボティクス)

Employing a learning system in which the "actor-critic' reinforcement learning method and a Gauss-sigmoid neural network are combined, a high performance learning system can be built due to the combination of the autonomy for reinforcement learning and the generalization capability for neural networks. Much time is needed when the conventional reinforcement learning method is used in order to train a robot with huge number of states such as our 1-legged hopping robot with 3-DOF. In this study, based on a simulation study for a wheel type inverted pendulum, we examined a few learning techniques different from conventional methods from the viewpoint of the improvement of learning speed. The obtained result shows that the increase of the number of modification steps as for connection strengths in neural networks is very effective for the decrease of learning cost.

Reinforcement Learning with Approximation Spaces

This paper introduces a rough set approach to reinforcement learning by swarms of cooperating agents. The problem considered in this paper is how to guide reinforcement learning based on knowledge of acceptable behavior patterns. This is made possible by considering behavior patterns of swarms in the context of approximation spaces. Rough set theory introduced by Zdzislaw Pawlak in the early 1980s provides a ground for deriving pattern-based rewards within approximation spaces. Both conventional and approximation space-based forms of reinforcement comparison and the actor-critic method as well as two forms of the off-policy Monte Carlo learning control method are investigated in this article. The study of swarm behavior by collections of biologically-inspired bots is carried out in the context of an artificial ecosystem testbed. This ecosystem has an ethological basis that makes it possible to observe and explain the behavior of biological organisms that carries over into the study of reinforcement learning by interacting robotic devices. The results of ecosystem experiments with six forms of reinforcement learning are given. The contribution of this article is the presentation of several viable alternatives to conventional reinforcement learning methods defined in the context of approximation spaces.