Natural Actor-Critic Algorithm Research Articles

Finite-Sample Analysis of Two-Time-Scale Natural Actor–Critic Algorithm

Actor-critic style two-time-scale algorithms are one of the most popular methods in reinforcement learning, and have seen great empirical success. However, their performance is not completely understood theoretically. In this paper, we characterize the <i>global</i> convergence of an online natural actor-critic algorithm in the tabular setting using a single trajectory of samples. Our analysis applies to very general settings, as we only assume ergodicity of the underlying Markov decision process. In order to ensure enough exploration, we employ an <inline-formula><tex-math notation="LaTeX">$\epsilon$</tex-math></inline-formula>-greedy sampling of the trajectory. For a fixed and small enough exploration parameter <inline-formula><tex-math notation="LaTeX">$\epsilon$</tex-math></inline-formula>, we show that the two-time-scale natural actor-critic algorithm has a rate of convergence of <inline-formula><tex-math notation="LaTeX">$\tilde{\mathcal {O}}(1/T^{1/4})$</tex-math></inline-formula>, where <inline-formula><tex-math notation="LaTeX">$T$</tex-math></inline-formula> is the number of samples, and this leads to a sample complexity of <inline-formula><tex-math notation="LaTeX">$\tilde{\mathcal {O}}(1/\delta ^{8})$</tex-math></inline-formula> samples to find a policy that is within an error of <inline-formula><tex-math notation="LaTeX">$\delta$</tex-math></inline-formula> from the <i>global optimum</i>. Moreover, by carefully decreasing the exploration parameter <inline-formula><tex-math notation="LaTeX">$\epsilon$</tex-math></inline-formula> as the iterations proceed, we present an improved sample complexity of <inline-formula><tex-math notation="LaTeX">$\tilde{\mathcal {O}}(1/\delta ^{6})$</tex-math></inline-formula> for convergence to the global optimum.

A Two-Timescale Stochastic Algorithm Framework for Bilevel Optimization: Complexity Analysis and Application to Actor-Critic

This paper analyzes a two-timescale stochastic algorithm framework for bilevel optimization. Bilevel optimization is a class of problems which exhibits a two-level structure, and its goal is to minimize an outer objective function with variables which are constrained to be the optimal solution to an (inner) optimization problem. We consider the case when the inner problem is unconstrained and strongly convex, while the outer problem is constrained and has a smooth objective function. We propose a two-timescale stochastic approximation (TTSA) algorithm for tackling such a bilevel problem. In the algorithm, a stochastic gradient update with a larger step size is used for the inner problem, while a projected stochastic gradient update with a smaller step size is used for the outer problem. We analyze the convergence rates for the TTSA algorithm under various settings: when the outer problem is strongly convex (resp. weakly convex), the TTSA algorithm finds an -optimal (resp. -stationary) solution, where is the total iteration number. As an application, we show that a two-timescale natural actor-critic proximal policy optimization algorithm can be viewed as a special case of our TTSA framework. Importantly, the natural actor-critic algorithm is shown to converge at a rate of in terms of the gap in expected discounted reward compared to a global optimal policy.

Robot Manipulation Skills Transfer for Sim-to-Real in Unstructured Environments

Robot force control that needs to be customized for the robot structure in unstructured environments with difficult-to-tune parameters guarantees robots’ compliance and safe human–robot interaction in an increasingly expanding work environment. Although reinforcement learning provides a new idea for the adaptive adjustment of these parameters, the policy often needs to be trained from scratch when used in new robotics, even in the same task. This paper proposes the episodic Natural Actor-Critic algorithm with action limits to improve robot admittance control and transfer motor skills between robots. The motion skills learned by simple simulated robots can be applied to complex real robots, reducing the difficulty of training and time consumption. The admittance control ensures the realizability and mobility of the robot’s compliance in all directions. At the same time, the reinforcement learning algorithm builds up the environment model and realizes the adaptive adjustment of the impedance parameters during the robot’s movement. In typical robot contact tasks, motor skills are trained in a robot with a simple structure in simulation and used for a robot with a complex structure in reality to perform the same task. The real robot’s performance in each task is similar to the simulated robot’s in the same environment, which verifies the method’s effectiveness.

Finite-Sample Analysis of Off-Policy Natural Actor–Critic With Linear Function Approximation

In this paper, we develop a novel variant of natural actor-critic algorithm using off-policy sampling and linear function approximation, and establish a sample complexity of O(ε-3), outperforming all the previously known convergence bounds of such algorithms. In order to overcome the divergence due to deadly triad in off-policy policy evaluation under function approximation, we develop a critic that employs n-step TD-learning algorithm with a properly chosen n. We present finite-sample convergence bounds on this critic, which are of independent interest. Furthermore, we develop a variant of natural policy gradient under function approximation, with an improved convergence rate of O(1/T) after T iterations. Combining the finite sample bounds of the actor and the critic, we obtain an overall O(ε-3) sample complexity. Our results were derived solely based on the assumption that the behavior policy sufficiently explores the state-action space, which is a much lighter assumption compared to the related literature.

Policy oscillation is overshooting

A majority of approximate dynamic programming approaches to the reinforcement learning problem can be categorized into greedy value function methods and value-based policy gradient methods. The former approach, although fast, is well known to be susceptible to the policy oscillation phenomenon. We take a fresh view to this phenomenon by casting, within the context of non-optimistic policy iteration, a considerable subset of the former approach as a limiting special case of the latter. We explain the phenomenon in terms of this view and illustrate the underlying mechanism with artificial examples. We also use it to derive the constrained natural actor-critic algorithm that can interpolate between the aforementioned approaches. In addition, it has been suggested in the literature that the oscillation phenomenon might be subtly connected to the grossly suboptimal performance in the Tetris benchmark problem of all attempted approximate dynamic programming methods. Based on empirical findings, we offer a hypothesis that might explain the inferior performance levels and the associated policy degradation phenomenon, and which would partially support the suggested connection. Finally, we report scores in the Tetris problem that improve on existing dynamic programming based results by an order of magnitude.

A Survey of Actor-Critic Reinforcement Learning: Standard and Natural Policy Gradients

Policy-gradient-based actor-critic algorithms are amongst the most popular algorithms in the reinforcement learning framework. Their advantage of being able to search for optimal policies using low-variance gradient estimates has made them useful in several real-life applications, such as robotics, power control, and finance. Although general surveys on reinforcement learning techniques already exist, no survey is specifically dedicated to actor-critic algorithms in particular. This paper, therefore, describes the state of the art of actor-critic algorithms, with a focus on methods that can work in an online setting and use function approximation in order to deal with continuous state and action spaces. After starting with a discussion on the concepts of reinforcement learning and the origins of actor-critic algorithms, this paper describes the workings of the natural gradient, which has made its way into many actor-critic algorithms over the past few years. A review of several standard and natural actor-critic algorithms is given, and the paper concludes with an overview of application areas and a discussion on open issues.

Tunnel ventilation controller design using an RLS-based natural actor-critic algorithm

Tunnel ventilation systems provide drivers with a comfortable and safe driving environment by generating sufficient airflow and by diluting the concentration of noxious contaminants below an acceptable level. For that purpose, tunnel ventilation systems contain mechanical equipment such as jet-fans, blowers and dust collectors. These machines consume large amount of energy, therefore, it is necessary to have an efficient operating algorithm for tunnel ventilation in terms of energy savings and safe driving. In this paper, a new reinforcement learning (RL) method is applied as the control algorithm. In the process of formulating the reward of the tunnel ventilation system, which is a performance index to be maximized in the RL methodology, the following two objectives are of great interest: maintaining an adequate level of pollutants and minimizing power consumption. The RL control algorithm adopted in this research is based on an actor-critic architecture and natural gradient method. Due to its ability to achieve the truly steepest direction of gradients, the natural gradient method can be a promising route to improving the efficacy of the actor module. Also, the recursive least-squares (RLS) method is employed to the critic module in order to improve the efficiency by which data is used. Using actual data collected from an existing tunnel ventilation system, extensive simulation studies were performed. It was confirmed that the suggested algorithm achieved the desired control goals and, when compared to previously developed RL-based control algorithms, improved the performance considerably.

Natural actor–critic algorithms

We present four new reinforcement learning algorithms based on actor–critic, natural-gradient and function-approximation ideas, and we provide their convergence proofs. Actor–critic reinforcement learning methods are online approximations to policy iteration in which the value-function parameters are estimated using temporal difference learning and the policy parameters are updated by stochastic gradient descent. Methods based on policy gradients in this way are of special interest because of their compatibility with function-approximation methods, which are needed to handle large or infinite state spaces. The use of temporal difference learning in this way is of special interest because in many applications it dramatically reduces the variance of the gradient estimates. The use of the natural gradient is of interest because it can produce better conditioned parameterizations and has been shown to further reduce variance in some cases. Our results extend prior two-timescale convergence results for actor–critic methods by Konda and Tsitsiklis by using temporal difference learning in the actor and by incorporating natural gradients. Our results extend prior empirical studies of natural actor–critic methods by Peters, Vijayakumar and Schaal by providing the first convergence proofs and the first fully incremental algorithms.

Impedance Learning for Robotic Contact Tasks Using Natural Actor-Critic Algorithm

Compared with their robotic counterparts, humans excel at various tasks by using their ability to adaptively modulate arm impedance parameters. This ability allows us to successfully perform contact tasks even in uncertain environments. This paper considers a learning strategy of motor skill for robotic contact tasks based on a human motor control theory and machine learning schemes. Our robot learning method employs impedance control based on the equilibrium point control theory and reinforcement learning to determine the impedance parameters for contact tasks. A recursive least-square filter-based episodic natural actor-critic algorithm is used to find the optimal impedance parameters. The effectiveness of the proposed method was tested through dynamic simulations of various contact tasks. The simulation results demonstrated that the proposed method optimizes the performance of the contact tasks in uncertain conditions of the environment.

Natural Actor-Critic

In this paper, we suggest a novel reinforcement learning architecture, the Natural Actor-Critic. The actor updates are achieved using stochastic policy gradients employing Amari's natural gradient approach, while the critic obtains both the natural policy gradient and additional parameters of a value function simultaneously by linear regression. We show that actor improvements with natural policy gradients are particularly appealing as these are independent of coordinate frame of the chosen policy representation, and can be estimated more efficiently than regular policy gradients. The critic makes use of a special basis function parameterization motivated by the policy-gradient compatible function approximation. We show that several well-known reinforcement learning methods such as the original Actor-Critic and Bradtke's Linear Quadratic Q-Learning are in fact Natural Actor-Critic algorithms. Empirical evaluations illustrate the effectiveness of our techniques in comparison to previous methods, and also demonstrate their applicability for learning control on an anthropomorphic robot arm.