Adaptive Critic Research Articles

Dual-Loop Optimal Control of a Robot Manipulator and Its Application in Warehouse Automation

The goal of this work is to develop the next generation of coordinated optimal planning and control schemes for real-world robotic applications, with cost-effective intelligent robots that can safely and robustly perform the tasks at hand. This article presents a dual-loop optimal hierarchical control scheme for robotic manipulators consisting of outer and inner loops. The outer kinematic control loop in the operational space provides a joint velocity reference signal to the inner one. The kinematic control is formulated as a closed-loop optimal control approach to trajectory generation where the desired target (possibly time-varying) is obtained by acting upon the feedback from the actual state of the robot. The kinematic control is obtained using a closed-form analytic solution of the Hamilton–Jacobi–Bellman (HJB) equation. The proposed methodology defines the task in terms of the integral cost function, which results in a global optimal solution. The inner dynamic control loop consists of a neural network (NN)-based adaptive critic (AC) optimal tracking control scheme. The online NN approximator-based dynamic controller learns the infinite-horizon cost function related to inner loop error dynamics in continuous time and calculates the corresponding optimal control input to minimize the cost function forward in time. The stability and the performance of the proposed control scheme are shown theoretically via the Lyapunov approach and also verified experimentally using a 7-DOF Barrett WAM robot manipulator. The warehousing applications of the proposed dual-loop control scheme have been demonstrated experimentally in an exact warehouse setting. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This article was motivated by our previous experiences in the Amazon Robotics Challenge (ARC) competitions as a participant. Although the major lessons from those events revealed many important issues toward warehouse automation both from robotic vision and grasping aspects, the application aspects of control engineering to solution for these real-world problems could significantly benefit the today’s highly automated society. The intention of this article is to address this situation by designing a cost-effective robot manipulator control scheme by optimizing robot manipulation trajectories in the outer loop along with the actuator input in the inner loop. Therefore, we propose a dual-loop optimal control scheme for robotic manipulations under a cost-optimal framework with guaranteed stability proof via the Lyapunov approach. This results in smoother trajectories with cost-effective control actuator inputs than the state-of-the-art methods. We believe that this work can be beneficial for academic and industrial research to design more advanced and optimized solutions in this domain.

Approximate dynamic programming solutions with a single network adaptive critic for a class of nonlinear systems

Approximate dynamic programming (ADP) formulation implemented with an adaptive critic (AC)-based neural network (NN) structure has evolved as a powerful technique for solving the Hamilton-Jacobi-Bellman (HJB) equations. As interest in ADP and the AC solutions are escalating with time, there is a dire need to consider possible enabling factors for their implementations. A typical AC structure consists of two interacting NNs, which is computationally expensive. In this paper, a new architecture, called the ‘cost-function-based single network adaptive critic (J-SNAC)’ is presented, which eliminates one of the networks in a typical AC structure. This approach is applicable to a wide class of nonlinear systems in engineering. In order to demonstrate the benefits and the control synthesis with the J-SNAC, two problems have been solved with the AC and the J-SNAC approaches. Results are presented, which show savings of about 50% of the computational costs by J-SNAC while having the same accuracy levels of the dual network structure in solving for optimal control. Furthermore, convergence of the J-SNAC iterations, which reduces to a least-squares problem, is discussed; for linear systems, the iterative process is shown to reduce to solving the familiar algebraic Ricatti equation.

Asymptotic tracking by a reinforcement learning-based adaptive critic controller

Adaptive critic (AC) based controllers are typically discrete and/or yield a uniformly ultimately bounded stability result because of the presence of disturbances and unknown approximation errors. A continuous-time AC controller is developed that yields asymptotic tracking of a class of uncertain nonlinear systems with bounded disturbances. The proposed AC-based controller consists of two neural networks (NNs) — an action NN, also called the actor, which approximates the plant dynamics and generates appropriate control actions; and a critic NN, which evaluates the performance of the actor based on some performance index. The reinforcement signal from the critic is used to develop a composite weight tuning law for the action NN based on Lyapunov stability analysis. A recently developed robust feedback technique, robust integral of the sign of the error (RISE), is used in conjunction with the feedforward action neural network to yield a semiglobal asymptotic result. Experimental results are provided that illustrate the performance of the developed controller.

A single network adaptive critic (SNAC) architecture for optimal control synthesis for a class of nonlinear systems

Even though dynamic programming offers an optimal control solution in a state feedback form, the method is overwhelmed by computational and storage requirements. Approximate dynamic programming implemented with an Adaptive Critic (AC) neural network structure has evolved as a powerful alternative technique that obviates the need for excessive computations and storage requirements in solving optimal control problems. In this paper, an improvement to the AC architecture, called the "Single Network Adaptive Critic (SNAC)" is presented. This approach is applicable to a wide class of nonlinear systems where the optimal control (stationary) equation can be explicitly expressed in terms of the state and costate variables. The selection of this terminology is guided by the fact that it eliminates the use of one neural network (namely the action network) that is part of a typical dual network AC setup. As a consequence, the SNAC architecture offers three potential advantages: a simpler architecture, lesser computational load and elimination of the approximation error associated with the eliminated network. In order to demonstrate these benefits and the control synthesis technique using SNAC, two problems have been solved with the AC and SNAC approaches and their computational performances are compared. One of these problems is a real-life Micro-Electro-Mechanical-system (MEMS) problem, which demonstrates that the SNAC technique is applicable to complex engineering systems.