Continuous Action Reinforcement Learning Research Articles

Continuous Action Reinforcement Learning From a Mixture of Interpretable Experts.

Reinforcement learning (RL) has demonstrated its ability to solve high dimensional tasks by leveraging non-linear function approximators. However, these successes are mostly achieved by 'black-box' policies in simulated domains. When deploying RL to the real world, several concerns regarding the use of a 'black-box' policy might be raised. In order to make the learned policies more transparent, we propose in this paper a policy iteration scheme that retains a complex function approximator for its internal value predictions but constrains the policy to have a concise, hierarchical, and human-readable structure, based on a mixture of interpretable experts. Each expert selects a primitive action according to a distance to a prototypical state. A key design decision to keep such experts interpretable is to select the prototypical states from trajectory data. The main technical contribution of the paper is to address the challenges introduced by this non-differentiable prototypical state selection procedure. Experimentally, we show that our proposed algorithm can learn compelling policies on continuous action deep RL benchmarks, matching the performance of neural network based policies, but returning policies that are more amenable to human inspection than neural network or linear-in-feature policies.

Optimal Incentive Strategy in Cloud-Edge Integrated Demand Response Framework for Residential Air Conditioning Loads

In the residential demand response area, currently the incentive-based method (e.g., direct load control, DLC) may impair users’ comfort and autonomy, while the price-based method can hardly guarantee users’ engagements. This paper proposes an edge-cloud integrated demand response framework to achieve an effect-predictable residential demand response without harming users’ benefits. First, we combine the cloud-computing resource (cloud) and the home-installed smart thermostats (edges) to formulate an efficient, cost-effective, and data-secured infrastructure to implement the demand response program. Then, we model the demand response problem between the load aggregator and its served residential users as a bi-level optimization problem, and the key is for the load aggregator to find the optimal incentive strategy. To solve this problem, we introduce an RL algorithm, i.e., Continuous Action Reinforcement Learning Automata, to quickly obtain the optimal incentive strategy under an incomplete information scenario. Simulation results based on 136 real-world residential users in Austin area demonstrate that the proposed CEI-DR framework can increase the social welfare by about $8.6/h compared to the traditional DLC method during a normal DR event.

A Reinforcement-learning-based Bidding Strategy for Power Suppliers with Limited Information

The power market is a typical imperfectly competitive market where power suppliers gain higher profits through strategic bidding behaviors. Most existing studies assume that a power supplier is accessible to the sufficient market information to derive an optimal bidding strategy. However, this assumption may not be true in reality, particularly when a power market is newly launched. To help power suppliers bid with the limited information, a modified continuous action reinforcement learning automata algorithm is proposed. This algorithm introduces the discretization and Dyna structure into continuous action reinforcement learning automata algorithm for easy implementation in a repeated game. Simulation results verify the effectiveness of the proposed learning algorithm.

Continuous reinforcement learning based ramp jump control for single-track two-wheeled robots

Single-track two-wheeled robots have become an important research topic in recent years, owing to their simple structure, energy savings and ability to run on narrow roads. However, the ramp jump remains a challenging task. In this study, we propose to realize a single-track two-wheeled robot ramp jump. We present a control method that employs continuous action reinforcement learning techniques for single-track two-wheeled robot control. We design a novel reward function for reinforcement learning, optimize the dimensions of the action space, and enable training under the deep deterministic policy gradient algorithm. Finally, we validate the control method through simulation experiments and successfully realize the single-track two-wheeled robot ramp jump task. Simulation results validate that the control method is effective and has several advantages over high-dimension action space control, reinforcement learning control of sparse reward function and discrete action reinforcement learning control.

Reinforcement Learning Based Decision Making of Operational Indices in Process Industry Under Changing Environment

The plant-wide production process is composed of multiple unit processes, in which the operational indices of each unit process are assigned and adjusted according to product quality, yield, and actual operating modes. Due to the changing operational conditions of the production process, the operational indices cannot be effectively adjusted by most of the model-based methods or evolutionary computation. In this article, the decision making of operational indices is formulated as a continuous state, continuous action reinforcement learning (RL) problem and a model-free RL algorithm is proposed, which learns a decision policy to determine the operational indices according to the actual operational conditions. Different from the existing methods, this article presents a multiactor networks ensemble algorithm and an actor-critic framework with stochastic policy to avoid falling into local optimums. The relatively overall optimal policy is obtained by extracting the results of parallel training of multiactor networks, which guarantees the optimality of the obtained policy. In addition, by using the experience replay, it is particularly valuable to effectively deal with the problem that lacking of sampling data in the model-free RL. Simulation studies are conducted on actual data of a mineral processing plant and the results demonstrate the effectiveness of the proposed algorithm.

Multivariable System Identification Method Based on Continuous Action Reinforcement Learning Automata

In this work, a closed-loop identification method based on a reinforcement learning algorithm is proposed for multiple-input multiple-output (MIMO) systems. This method could be an attractive alternative solution to the problem that the current frequency-domain identification algorithms are usually dependent on the attenuation factor. With this method, after continuously interacting with the environment, the optimal attenuation factor can be identified by the continuous action reinforcement learning automata (CARLA), and then the corresponding parameters could be estimated in the end. Moreover, the proposed method could be applied to time-varying systems online due to its online learning ability. The simulation results suggest that the presented approach can meet the requirement of identification accuracy in both square and non-square systems.

A novel dynamics model of ball-screw feed drives based on theoretical derivations and deep learning

High fidelity models of feed drive are critical factors to increase positioning accuracy and decrease contour error. To predict feed drives dynamics, this paper reports a novel method for modeling dynamics of feed drive by combining advantages of theoretical derivations and deep learning. First, the paper derives a rigid-flexible-combined dynamics model (RFCDM) for feed drive from classical dynamics theory. Then parameters identification of RFCDM is accomplished by referring product manuals and conducting constant velocity experiment with different feed rates. Continuous action reinforcement learning automata (CARLA) is adopted to tune all parameters of RFCDM simultaneously. A simulation error estimation model (SEEM) is applied to approximate simulation error between models simulation position and worktables actual position. The hybrid dynamics model (HDM) of feed drives which integrates RFCDM with SEEM is validated by experiments with various trajectories. Experimental results show that the gap between HDMs prediction position and worktables actual position is on the order of magnitude of 0.01 mm which is about 1/10 of the tracking error, indicating the HDM can predict the dynamics of feed drives with safe accuracy.

Probabilistic Policy Reuse for Safe Reinforcement Learning

This work introduces Policy Reuse for Safe Reinforcement Learning , an algorithm that combines Probabilistic Policy Reuse and teacher advice for safe exploration in dangerous and continuous state and action reinforcement learning problems in which the dynamic behavior is reasonably smooth and the space is Euclidean. The algorithm uses a continuously increasing monotonic risk function that allows for the identification of the probability to end up in failure from a given state. Such a risk function is defined in terms of how far such a state is from the state space known by the learning agent. Probabilistic Policy Reuse is used to safely balance the exploitation of actual learned knowledge, the exploration of new actions, and the request of teacher advice in parts of the state space considered dangerous. Specifically, the π-reuse exploration strategy is used. Using experiments in the helicopter hover task and a business management problem, we show that the π-reuse exploration strategy can be used to completely avoid the visit to undesirable situations while maintaining the performance (in terms of the classical long-term accumulated reward) of the final policy achieved.

A reinforcement learning approach to coordinate exploration with limited communication in continuous action games

AbstractLearning automata are reinforcement learners belonging to the class of policy iterators. They have already been shown to exhibit nice convergence properties in a wide range of discrete action game settings. Recently, a new formulation for a continuous action reinforcement learning automata (CARLA) was proposed. In this paper, we study the behavior of these CARLA in continuous action games and propose a novel method for coordinated exploration of the joint-action space. Our method allows a team of independent learners, using CARLA, to find the optimal joint action in common interest settings. We first show that independent agents using CARLA will converge to a local optimum of the continuous action game. We then introduce a method for coordinated exploration which allows the team of agents to find the global optimum of the game. We validate our approach in a number of experiments.

Continuous action reinforcement learning for control-affine systems with unknown dynamics

Continuous action reinforcement learning for control-affine systems with unknown dynamics

Combination of Continuous Action Reinforcement Learning Automata & PSO to Design a PID Controller for AVR System

Combination of Continuous Action Reinforcement Learning Automata & PSO to Design a PID Controller for AVR System

Power System Dynamic Stability Enhancement Using Optimum Design of PSS and Static Phase Shifter Based Stabilizer

Power system stability enhancement via individual and coordinated optimum design of power system stabilizer and static phase shifter based stabilizer is thoroughly investigated in this paper. The coordinated design problem of excitation and static phase shifter-based stabilizer at nominal loading condition over a wide range of uncertainties and system parameter variations is formulated as an optimization problem with an eigenvalue–based objective function. The reinforcement learning automata based optimization algorithm called Combinatorial Discrete and Continuous Action Reinforcement Learning Automata (CDCARLA) is employed to search for the optimal setting of the proposed controller parameters. The participation factors method is used to identify the poorly damped electromechanical mode which is used in the evaluation of the eigenvalue–based objective function. This study also presents a singular value decomposition–based approach to assess and measure the controllability of the identified electromechanical mode by different control inputs over a wide range of loading conditions. For evaluation of the effectiveness and the robustness of the proposed stabilizers, a weakly connected power system with different disturbances, loading conditions, and system parameter variations is considered. The nonlinear simulation results and the eigenvalues analysis indicate a high performance of the proposed stabilizers and their ability to provide efficient damping of the low frequency oscillations.

Continuous-action reinforcement learning with fast policy search and adaptive basis function selection

As an important approach to solving complex sequential decision problems, reinforcement learning (RL) has been widely studied in the community of artificial intelligence and machine learning. However, the generalization ability of RL is still an open problem and it is difficult for existing RL algorithms to solve Markov decision problems (MDPs) with both continuous state and action spaces. In this paper, a novel RL approach with fast policy search and adaptive basis function selection, which is called Continuous-action Approximate Policy Iteration (CAPI), is proposed for RL in MDPs with both continuous state and action spaces. In CAPI, based on the value functions estimated by temporal-difference learning, a fast policy search technique is suggested to search for optimal actions in continuous spaces, which is computationally efficient and easy to implement. To improve the generalization ability and learning efficiency of CAPI, two adaptive basis function selection methods are developed so that sparse approximation of value functions can be obtained efficiently both for linear function approximators and kernel machines. Simulation results on benchmark learning control tasks with continuous state and action spaces show that the proposed approach not only can converge to a near-optimal policy in a few iterations but also can obtain comparable or even better performance than Sarsa-learning, and previous approximate policy iteration methods such as LSPI and KLSPI.

APPLICATION OF NOVEL REINFORCEMENT LEARNING AUTOMATA APPROACH IN POWER SYSTEM REGULATION

In this paper, a novel efficient optimization method based on reinforcement learning automata (RLA) for optimum parameters setting of conventional proportional-integral-derivative (PID) controller for AVR system of power synchronous generator is proposed. The proposed method is Combinatorial Discrete and Continuous Action Reinforcement Learning Automata (CDCARLA) which is able to explore and learn to improve control performance without the knowledge of the analytical system model. This paper demonstrates the full details of the CDCARLA technique and compares its performance with Particle Swarm Optimization (PSO) as an efficient evolutionary optimization method. The proposed method has been applied to PID controller design. The simulation results show the superior efficiency and robustness of the proposed method.

Parameter optimization of multimachine power system conventional stabilizers using CDCARLA method

In this paper, a novel Combinatorial Discrete and Continuous Action Reinforcement Learning Automata (CDCARLA) based approach for optimal design of multimachine power system stabilizers (PSSs) is presented. The proposed CDCARLA based design approach is a combined procedure of two optimization stages in discrete and continuous spaces for fast convergence and high optimization efficiency. The potential of the proposed approach in seeking the optimal settings of the widely used conventional lead-lag PSSs’ parameters is investigated and assessed in multimachine power systems. The performance and robustness of the proposed CDCARLA based PSS is evaluated under different power system disturbances. The performance of the proposed stabilizer is also compared with other stabilizers reported in the literature including the multi-band PSSs for a two-area four-machine power system. Simulation results show the effectiveness and robustness of the proposed CDCARLA PSS in damping local and inter area oscillation modes under various disturbances, and confirm its superiority in comparison with other types of PSSs.

Continuous action reinforcement learning automata and their application to adaptive digital filter design

In the design of adaptive IIR filters, the multi-modal nature of the error surfaces can limit the use of gradient-based and other iterative search methods. Stochastic learning automata have previously been shown to have global optimisation properties making them suitable for the optimisation of filter coefficients. Continuous action reinforcement learning automata are presented as an extension to the standard automata which operate over discrete parameter sets. Global convergence is claimed, and demonstrations are carried out via a number of computer simulations.

On-line PID tuning for engine idle-speed control using continuous action reinforcement learning automata

PID systems are widely used to apply control without the need to obtain a dynamic model. However, the performance of controllers designed using standard on-line tuning methods, such as Ziegler–Nichols, can often be significantly improved. In this paper the tuning process is automated through the use of continuous action reinforcement learning automata (CARLA). These are used to simultaneously tune the parameters of a three term controller on-line to minimise a performance objective. Here the method is demonstrated in the context of engine idle-speed control; the algorithm is first applied in simulation on a nominal engine model, and this is followed by a practical study using a Ford Zetec engine in a test cell. The CARLA provides marked performance benefits over a comparable Ziegler–Nichols tuned controller in this application.

Continuous action reinforcement learning applied to vehicle suspension control

A new reinforcement learning algorithm is introduced which can be applied over a continuous range of actions. The learning algorithm is reward-inaction based, with a set of probability density functions being used to determine the action set. An experimental study is presented, based on the control of a semi-active suspension system on a road-going, four wheeled, passenger vehicle. The control objective is to minimise the mean square acceleration of the vehicle body, thus improving the ride isolation qualities of the vehicle. This represents a difficult class of learning problems, owing to the stochastic nature of the road input disturbance together with unknown high order dynamics, sensor noise and the non-linear (semi-active) control actuators. The learning algorithm described here operates over a bounded continuous action set, is robust to high levels of noise and is ideally suited to operating in a parallel computing environment.