Actor-critic Learning Research Articles

Multiagent Soft Actor–Critic Learning for Distributed ESS Enabled Robust Voltage Regulation of Active Distribution Grids

Multiagent Soft Actor–Critic Learning for Distributed ESS Enabled Robust Voltage Regulation of Active Distribution Grids

Distribution network voltage control considering virtual power plants cooperative optimization with transactive energy

The integration of massive distributed energy resources (DERs) brings new challenges for the distribution networks (DNs) operation control. To control DERs effectively, virtual power plants (VPPs) are introduced into DNs. Conventionally, the control strategy of VPPs is formulated as an optimization model without considering voltage control and transactive energy (TE) in DNs. In this article, multiple VPPs cooperative optimization with TE is studied and cast as a nonlinear programming model. It aims to optimize the voltage control of DNs and the operation profits of VPPs. To dynamically capture the optimal operation state of VPPs in DNs, the distribution locational marginal pricing and DN partition are incorporated into the model. In general, this type of model is computationally challenging and mainly solved by approximate algorithms with low efficiency. Recently, multi-agent deep reinforcement learning (MADRL) is emerging as a scalable and promising data-driven approach. We propose an augmented MADRL solution to this problem. In the training stage, one soft actor-critic learning agent augmented by parameter sharing strategy is employed to simultaneously capture the state-action patterns of multiple VPPs. The prioritized experience replay technique is applied to improve learning efficiency and speed. Numerical test results demonstrate the superiority of the proposed method over benchmark methods.

GRU-integrated constrained soft actor-critic learning enabled fully distributed scheduling strategy for residential virtual power plant

In this study, a novel residential virtual power plant (RVPP) scheduling method that leverages a gate recurrent unit (GRU)-integrated deep reinforcement learning (DRL) algorithm is proposed. In the proposed scheme, the GRU- integrated DRL algorithm guides the RVPP to participate effectively in both the day-ahead and real-time markets, lowering the electricity purchase costs and consumption risks for end-users. The Lagrangian relaxation technique is introduced to transform the constrained Markov decision process (CMDP) into an unconstrained optimization problem, which guarantees that the constraints are strictly satisfied without determining the penalty coefficients. Furthermore, to enhance the scalability of the constrained soft actor-critic (CSAC)-based RVPP scheduling approach, a fully distributed scheduling architecture was designed to enable plug-and-play in the residential distributed energy resources (RDER). Case studies performed on the constructed RVPP scenario validated the performance of the proposed methodology in enhancing the responsiveness of the RDER to power tariffs, balancing the supply and demand of the power grid, and ensuring customer comfort.

Bayesian Strategy Networks Based Soft Actor-Critic Learning

A strategy refers to the rules that the agent chooses the available actions to achieve goals. Adopting reasonable strategies is challenging but crucial for an intelligent agent with limited resources working in hazardous, unstructured, and dynamic environments to improve the system’s utility, decrease the overall cost, and increase mission success probability. This article proposes a novel hierarchical strategy decomposition approach based on Bayesian chaining to separate an intricate policy into several simple sub-policies and organize their relationships as Bayesian strategy networks (BSN). We integrate this approach into the state-of-the-art DRL method—soft actor-critic (SAC), and build the corresponding Bayesian soft actor-critic (BSAC) model by organizing several sub-policies as a joint policy. Our method achieves the state-of-the-art performance on the standard continuous control benchmarks in the OpenAI Gym environment. The results demonstrate that the promising potential of the BSAC method significantly improves training efficiency. Furthermore, we extend the topic to the Multi-Agent systems (MAS), discussing the potential research fields and directions.

Joint Computing, Pushing, and Caching Optimization for Mobile-Edge Computing Networks via Soft Actor–Critic Learning

Joint Computing, Pushing, and Caching Optimization for Mobile-Edge Computing Networks via Soft Actor–Critic Learning

Expert System-Based Multiagent Deep Deterministic Policy Gradient for Swarm Robot Decision Making.

In this article, an expert system-based multiagent deep deterministic policy gradient (ESB-MADDPG) is proposed to realize the decision making for swarm robots. Multiagent deep deterministic policy gradient (MADDPG) is a multiagent reinforcement learning algorithm proposed to utilize a centralized critic within the actor-critic learning framework, which can reduce policy gradient variance. However, it is difficult to apply traditional MADDPG to swarm robots directly as it is time consuming during the path planning, rendering it necessary to propose a faster method to gather the trajectories. Besides, the trajectories obtained by the MADDPG are continuous by straight lines, which is not smooth and will be difficult for the swarm robots to track. This article aims to solve these problems by closing the above gaps. First, the ESB-MADDPG method is proposed to improve the training speed. The smooth processing of the trajectory is designed in the ESB-MADDPG. Furthermore, the expert system also provides us with many trained offline trajectories, which avoid the retraining each time we use the swarm robots. Considering the gathered trajectories, the model predictive control (MPC) algorithm is introduced to realize the optimal tracking of the offline trajectories. Simulation results show that combining ESB-MADDPG and MPC can realize swarm robot decision making efficiently.

Cooperative Finitely Excited Learning for Dynamical Games.

In this article, we propose a way to enhance the learning framework for zero-sum games with dynamics evolving in continuous time. In contrast to the conventional centralized actor-critic learning, a novel cooperative finitely excited learning approach is developed to combine the online recorded data with instantaneous data for efficiency. By using an experience replay technique for each agent and distributed interaction amongst agents, we are able to replace the classical persistent excitation condition with an easy-to-check cooperative excitation condition. This approach also guarantees the consensus of the distributed actor-critic learning on the solution to the Hamilton-Jacobi-Isaacs (HJI) equation. It is shown that both the closed-loop stability of the equilibrium point and convergence to the Nash equilibrium can be guaranteed. Simulation results demonstrate the efficacy of this approach compared to previous methods.

Reinforcement Learning-Based Adaptive Optimal Control for Nonlinear Systems With Asymmetric Hysteresis.

This article investigates the adaptive optimal tracking problem for a class of nonlinear affine systems with asymmetric Prandtl-Ishlinskii (PI) hysteresis nonlinearities based on actor-critic (A-C) learning mechanisms. Considering the huge obstacles arising from the uncertainty of hysteresis nonlinearity in actuators, we develop a scheme for the conflict between the construction of Hamilton functions and hysteresis nonlinearity. The actuator hysteresis forces the input into a hysteresis delay, thus preventing the Hamilton function from getting the current moment's input instantly and thus making optimization impossible. In the first step, an inverse model is constructed to compensate for the hysteresis model with a shift factor. In the second step, we compensate for the control input by designing a feedback controller and incorporating the estimation and approximation errors into the Hamilton error. Optimal control, the other part of the actual control input, is obtained by taking partial derivatives of the Hamiltonian function after the nonlinearities have been circumvented. At the end, a simulation is given to validate the developed solution.

Attention-Enhanced Actor–Critic Learning for Household Nonintrusive Load Monitoring

Attention-Enhanced Actor–Critic Learning for Household Nonintrusive Load Monitoring

Sample-Observed Soft Actor-Critic Learning for Path Following of a Biomimetic Underwater Vehicle

This paper addresses a learning-based path following control scheme for a biomimetic underwater vehicle (BUV) driven by undulatory fins. A dynamic line-of-sight (DLOS) guidance system is designed, which uses a virtual ball with a dynamic radius to detect the reference path. This DLOS system guides our BUV in the path following control and extracts essential information for the Markov decision process (MDP) of the control task. A deep reinforcement learning (DRL) algorithm, sample-observed soft actor-critic (SOSAC) is proposed. The can train out control policy with greater cumulative reward and higher success rate by using two tricks: sample observation and sample diversification. Based on the DLOS system, the MDP of the control task, and a multilayer perceptron (MLP) trained by the SOSAC, our control scheme is established. Experiments show that our BUV can successfully achieve path following control in an indoor pool environment by using this control scheme. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —The motivation of this paper is to design a practical end-to-end path following control scheme for the BUV driven by undulatory fins, and verify this scheme in a real-world environment. Unlike common autonomous underwater vehicles (AUVs) using axial propellers, the BUVs apply biomimetic propellers such as the undulatory fin. Multimodel wave patterns can be implemented by the undulatory fin, which generates nonlinear thrust and lateral force simultaneously. This propulsive feature makes the driving force on different directions of the BUV to be strong coupled, and it is complicated to convert the outputs of a common controller into waveform parameters of the undulatory fins to control the BUV. Therefore, in this paper, we proposed an end-to-end learning-based path following controller, which observes environmental information and directly generates waveform parameters to control our BUV. Experiments suggest that our control scheme is practical and valid.

Kernel-based Actor-Critic Learning Framework for Autonomous Brain Control on Trajectory

Kernel-based Actor-Critic Learning Framework for Autonomous Brain Control on Trajectory

Reinforcement actor-critic learning as a rehearsal in MicroRTS

Abstract Real-time strategy (RTS) games have provided a fertile ground for AI research with notable recent successes based on deep reinforcement learning (RL). However, RL remains a data-hungry approach featuring a high sample complexity. In this paper, we focus on a sample complexity reduction technique called reinforcement learning as a rehearsal (RLaR) and on the RTS game of MicroRTS to formulate and evaluate it. RLaR has been formulated in the context of action-value function based RL before. Here, we formulate it for a different RL framework, called actor-critic RL. We show that on the one hand the actor-critic framework allows RLaR to be much simpler, but on the other hand, it leaves room for a key component of RLaR–a prediction function that relates a learner’s observations with that of its opponent. This function, when leveraged for exploration, accelerates RL as our experiments in MicroRTS show. Further experiments provide evidence that RLaR may reduce actor noise compared to a variant that does not utilize RLaR’s exploration. This study provides the first evaluation of RLaR’s efficacy in a domain with a large strategy space.

Multi-Timescale Actor-Critic Learning for Computing Resource Management With Semi-Markov Renewal Process Mobility

Multi-Timescale Actor-Critic Learning for Computing Resource Management With Semi-Markov Renewal Process Mobility

Actor–critic learning based PID control for robotic manipulators

In this paper, we propose a reinforcement learning structure for auto-tuning PID gains by solving an optimal tracking control problem for robot manipulators. Capitalizing on the actor–critic framework implemented by neural networks, we achieve optimal tracking performance, estimating unknown system dynamics simultaneously. The critic network is used to approximate the cost function, which serves as an indicator of control performance. With feedback from the critic, the actor network learns time-varying PID gains over time to optimize control input, thereby steering the system toward optimal performance. Furthermore, we utilize Lyapunov’s direct method to demonstrate the stability of the closed-loop system. This approach provides an analytical procedure for a stable robot manipulator system to systematically adjust PID gains, bypassing the ad-hoc and painstaking process. The resultant actor–critic PID-like control exhibits stable adaptive and learning capabilities while maintaining a simple structure and inexpensive online computational demands. Numerical simulations underscore the effectiveness and advantages of the proposed actor–critic neural network PID control.

Adaptive fault-tolerant control for non-minimum phase hypersonic vehicles based on adaptive dynamic programming

In this paper, a novel adaptive Fault-Tolerant Control (FTC) strategy is proposed for non-minimum phase Hypersonic Vehicles (HSVs) that are affected by actuator faults and parameter uncertainties. The strategy is based on the output redefinition method and Adaptive Dynamic Programming (ADP). The intelligent FTC scheme consists of two main parts: a basic fault-tolerant and stable controller and an ADP-based supplementary controller. In the basic FTC part, an output redefinition approach is designed to make zero-dynamics stable with respect to the new output. Then, Ideal Internal Dynamic (IID) is obtained using an optimal bounded inversion approach, and a tracking controller is designed for the new output to realize output tracking of the non-minimum phase HSV system. For the ADP-based compensation control part, an Action-Dependent Heuristic Dynamic Programming (ADHDP) adopting an actor-critic learning structure is utilized to further optimize the tracking performance of the HSV control system. Finally, simulation results are provided to verify the effectiveness and efficiency of the proposed FTC algorithm.

Hierarchical Multiagent Formation Control Scheme via Actor-Critic Learning.

This article presents a nearly optimal solution to the cooperative formation control problem for large-scale multiagent system (MAS). First, multigroup technique is widely used for the decomposition of the large-scale problem, but there is no consensus between different subgroups. Inspired by the hierarchical structure applied in the MAS, a hierarchical leader-following formation control structure with multigroup technique is constructed, where two layers and three types of agents are designed. Second, adaptive dynamic programming technique is conformed to the optimal formation control problem by the establishment of performance index function. Based on the traditional generalized policy iteration (PI) algorithm, the multistep generalized policy iteration (MsGPI) is developed with the modification of policy evaluation. The novel algorithm not only inherits the advantages of high convergence speed and low computational complexity in the generalized PI algorithm but also further accelerates the convergence speed and reduces run time. Besides, the stability analysis, convergence analysis, and optimality analysis are given for the proposed multistep PI algorithm. Afterward, a neural network-based actor-critic structure is built for approximating the iterative control policies and value functions. Finally, a large-scale formation control problem is provided to demonstrate the performance of our developed hierarchical leader-following formation control structure and MsGPI algorithm.

Reinforcement learning-based saturated adaptive robust output-feedback funnel control of surface vessels in different weather conditions

This article tackles the problem of saturated adaptive robust actor–critic learning PID tracking control design guaranteeing a prescribed funnel performance for ships with multiple dead bands in which the ship’s velocities/accelerations and dynamic parameters are unknown and time-variant subject to the availability of external disturbances. The controller is proposed by integrating an observer, adaptive actor–critic multi-layer neural networks, generalized saturation functions, the prescribed performance funnel control, an adaptive robust controller and multiple dead bands. The observer calculates the estimation of ship’s velocities/accelerations, in which the peaking phenomenon is prevented by limiting the observer signals. Within the structure of adaptive actor–critic design, adaptive multi-layer neural networks are exerted, which counteracts against ship’s time-variant uncertainties and the actuator saturation nonlinearity. In order to reduce the likelihood of actuator saturation well, generalized saturation functions are taken into account. To realize prespecified transient and steady-state performance of the tracking errors, the prescribed performance control is applied. With the purpose of avoiding the corner cutting to increase the trajectory tracking accuracy, curvature of the circular trajectories is calculated and the angle of arc is compensated. An adaptive robust controller is utilized to resist against external disturbances, and a scheme of multiple dead bands is devised in this paper to protect the actuator system in heavy seas.

Koopman-operator-based learning control of air-breathing hypersonic vehicles with nonminimum phase properties

In this article, a prescribed learning controller is developed for nonminimum phase air-breathing hypersonic vehicles (AHVs) in the presence of parametric uncertainties and external disturbances. In comparison with the current state of the art, the most significant feature of our control design lies in introducing the Koopman operator to construct an intelligent output redefinition to overcome the nonminimum phase behavior. To evaluate the current control behavior and enhance the learning ability, an improved continuous-time performance index is developed under the actor–critic learning structure. Furthermore, combined with a nonlinear disturbance observer, prescribed learning control is proposed to design control commands, while compensating for lumped disturbances, including neural approximation errors and external disturbances. Numerical simulations have been performed to highlight the superiority of the proposed learning control.

Receding Horizon Actor–Critic Learning Control for Nonlinear Time-Delay Systems With Unknown Dynamics

With the development of modern mechatronics and networked systems, the controller design of time-delay systems has received notable attention. Time delays can greatly influence the stability and performance of the systems, especially for optimal control design. In this article, we propose a receding horizon actor–critic learning control approach for near-optimal control of nonlinear time-delay systems (RACL-TD) with unknown dynamics. In the proposed approach, a data-driven predictor for nonlinear time-delay systems is first learned based on the Koopman theory using precollected samples. Then, a receding horizon actor–critic architecture is designed to learn a near-optimal control policy. In RACL-TD, the terminal cost is determined by using the Lyapunov–Krasovskii approach so that the influences of the delayed states and control inputs can be well addressed. Furthermore, a relaxed terminal condition is present to reduce the computational cost. The convergence and optimality of RACL-TD in each prediction interval as well as the closed-loop property of the system are discussed and analyzed. Simulation results on a two-stage time-delayed chemical reactor illustrate that RACL-TD can achieve better control performance than nonlinear model predictive control (MPC) and infinite-horizon adaptive dynamic programming. Moreover, RACL-TD can have less computational cost than nonlinear MPC.

Federated Multiagent Actor–Critic Learning Task Offloading in Intelligent Logistics

Intelligent logistics empowered by artificial intelligence (AI) has become an inevitable trend in the development of modern logistics, thus a convenient and efficient logistics system has attracted widespread attention. However, how to use AI to execute computation-intensive applications on resource-constrained logistics vehicles still faces enormous challenges. For the dependent applications in intelligent logistics, this paper investigates a task dynamic offloading strategy for logistics vehicles-edge collaboration with multiple dependent tasks, considering the inter-task dependency, to guarantee the quality of service (QoS) requirements of logistics vehicles. Firstly, the dependent application ARCore is modeled and transformed into a model with a linear execution sequence. Then, based on this task model, the joint task offloading and resource allocation problem is formulated. The goal is to minimize the weighted sum cost of the execution delay and energy consumption while guaranteeing the delay tolerance and computing resource constraints of the tasks. Furthermore, we propose a federated logistics vehicles-edge collaborative computation (FECC) offloading framework to solve the optimization problem, which only requires each agent to share its model parameters without sharing local training data, thereby reducing the computation complexity and signaling overhead of the multi-agent training process. Numerical results show that the proposed strategy has significant advantages in terms of total system cost compared to the baseline strategy.