Actor-critic Structure Research Articles

Multi-agent DRL for edge computing: A real-time proportional compute offloading

In the Industrial Internet of Things, devices with limited computing power and energy storage often rely on offloading tasks to edge servers for processing. However, existing methods are plagued by the high cost of device communication and unstable training processes. Consequently, Deep reinforcement learning (DRL) has emerged as a promising solution to tackle the computation offloading problem. In this paper, we propose a framework called multi-agent twin delayed shared deep deterministic policy gradient algorithm (MASTD3) based on DRL. Firstly, we formulate the task offloading conundrum as a long-term optimization problem, which aids in mitigating the challenge of deciding between local or remote task execution by a device, leading to more effective task offloading management. Secondly, we enhance MASTD3 by introducing a priority experience replay buffer mechanism and a model sample replay buffer mechanism, thus improving sample utilization and overcoming the cold-start problem associated with long-term optimization. Moreover, we refine the actor-critic structure, enabling all agents to share the same critic network. This modification accelerates convergence speed during the training process and reduces computational costs during runtime. Finally, experimental results demonstrate that MASTD3 effectively addresses the proportional offloading problem, which is optimized by 44.32%, 29.26%, and 17.47% compared to DDPQN, MADDPG, and FLoadNet.

Adaptive dynamic programming base on MMC device of a flexible high-altitude long endurance aircraft

Flexible high-altitude long endurance (HALE) aircraft face challenges such as the delay effect caused by aeroelasticity and low efficiency during high-altitude cruising. This paper addresses a new active control scheme that replaces traditional aerodynamic control surfaces with longitudinal and lateral moving masses and adopts an observer-based adaptive dynamic programming (ADP) control strategy, aiming to solve the attitude control of HALE aircraft with flexible wings. Firstly, considering the account of unsteady aerodynamics, attitude angular velocity, and moving masses, complete dynamic model is established for a flexible aircraft, in order to more accurately describe the state of the HALE aircraft. This model includes two moving masses to replace traditional ailerons and elevators. It is difficult to design attitude control strategies considering the delay effects and unknown dynamic disturbances of HALE aircraft. To this end, the actor-critic structure of the ADP approach is designed to achieve the near-optimal control strategy of the system, which eliminates the need for prior knowledge of model parameters. A fixed time disturbance observer (FTDO) is proposed to estimate the future tracking error information and unknown disturbances to improve the control performance of the attitude. Finally, the stability of the system is proved via the Lyapunov technique and a comprehensive simulation of HALE aircraft is provided. The simulation results show that the proposed control algorithm can enable the aircraft attitude angles to quickly response and maintain stability under gust wind. Compared with a single ADP control strategy, this paper's algorithm has advantages in response speed and error.

Data-Driven Optimal Bipartite Consensus Control for Second-Order Multiagent Systems via Policy Gradient Reinforcement Learning.

This article investigates the optimal bipartite consensus control (OBCC) problem for unknown second-order discrete-time multiagent systems (MASs). First, the coopetition network is constructed to describe the cooperative and competitive relationships between agents, and the OBCC problem is proposed by the tracking error and related performance index function. Based on the distributed policy gradient reinforcement learning (RL) theory, a data-driven distributed optimal control strategy is obtained to guarantee the bipartite consensus of all agents' position and velocity states. In addition, the offline data sets ensure the learning efficiency of the system. These data sets are generated by running the system in real time. Besides, the designed algorithm is an asynchronous version, which is essential to solve the challenge caused by the computational ability difference between nodes in MASs. Then, by means of the functional analysis and Lyapunov theory, the stability of the proposed MASs and the convergence of the learning process are analyzed. Furthermore, an actor-critic structure containing two neural networks is used to implement the proposed methods. Finally, a numerical simulation shows the effectiveness and validity of the results.

A Hybrid Controller for Musculoskeletal Robots Targeting Lifting Tasks in Industrial Metaverse.

In manufacturing, musculoskeletal robots have gained more attention with the potential advantages of flexibility, robustness, and adaptability over conventional serial-link rigid robots. Focusing on the fundamental lifting tasks, a hybrid controller is proposed to overcome control challenges of such robots for widely applications in industry. The metaverse technology offers an available simulated-reality-based platform to verify the proposed method. The hybrid controller contains two main parts. A muscle-synergy-based radial basis function (RBF) network is proposed as the feedforward controller, which is able to characterize the phasic and the tonic muscle synergies simultaneously. The adaptive dynamic programming (ADP) is applied as the feedback controller to address the optimal control problem. The actor-critic structure is applied in the ADP-based controller, where the critic network is trained to approximate the optimal performance index and the actor network is trained to compute the optimal muscle excitations. Furthermore, the convergence and stability of the ADP algorithm are also analyzed. Finally, experiments have been designed to verify the effectiveness of this hybrid controller on an upper limb musculoskeletal system, and the comparisons with other controllers are also illustrated. The results show that the proposed controller can obtain a satisfactory performance for lifting tasks.

Nearly Optimal Control for Mixed Zero-Sum Game Based on Off-Policy Integral Reinforcement Learning.

In this article, we solve a class of mixed zero-sum game with unknown dynamic information of nonlinear system. A policy iterative algorithm that adopts integral reinforcement learning (IRL), which does not depend on system information, is proposed to obtain the optimal control of competitor and collaborators. An adaptive update law that combines critic-actor structure with experience replay is proposed. The actor function not only approximates optimal control of every player but also estimates auxiliary control, which does not participate in the actual control process and only exists in theory. The parameters of the actor-critic structure are simultaneously updated. Then, it is proven that the parameter errors of the polynomial approximation are uniformly ultimately bounded. Finally, the effectiveness of the proposed algorithm is verified by two given simulations.

Reinforcement learning-based adaptive motion control for autonomous vehicles via actor-critic structure

Reinforcement learning-based adaptive motion control for autonomous vehicles via actor-critic structure

Realizing asynchronous finite-time robust tracking control of switched flight vehicles by using nonfragile deep reinforcement learning.

In this study, a novel nonfragile deep reinforcement learning (DRL) method was proposed to realize the finite-time control of switched unmanned flight vehicles. Control accuracy, robustness, and intelligence were enhanced in the proposed control scheme by combining conventional robust control and DRL characteristics. In the proposed control strategy, the tracking controller consists of a dynamics-based controller and a learning-based controller. The conventional robust control approach for the nominal system was used for realizing a dynamics-based baseline tracking controller. The learning-based controller based on DRL was developed to compensate model uncertainties and enhance transient control accuracy. The multiple Lyapunov function approach and mode-dependent average dwell time approach were combined to analyze the finite-time stability of flight vehicles with asynchronous switching. The linear matrix inequalities technique was used to determine the solutions of dynamics-based controllers. Online optimization was formulated as a Markov decision process. The adaptive deep deterministic policy gradient algorithm was adopted to improve efficiency and convergence. In this algorithm, the actor-critic structure was used and adaptive hyperparameters were introduced. Unlike the conventional DRL algorithm, nonfragile control theory and adaptive reward function were used in the proposed algorithm to achieve excellent stability and training efficiency. We demonstrated the effectiveness of the presented algorithm through comparative simulations.

Hierarchical Multiagent Formation Control Scheme via Actor-Critic Learning.

This article presents a nearly optimal solution to the cooperative formation control problem for large-scale multiagent system (MAS). First, multigroup technique is widely used for the decomposition of the large-scale problem, but there is no consensus between different subgroups. Inspired by the hierarchical structure applied in the MAS, a hierarchical leader-following formation control structure with multigroup technique is constructed, where two layers and three types of agents are designed. Second, adaptive dynamic programming technique is conformed to the optimal formation control problem by the establishment of performance index function. Based on the traditional generalized policy iteration (PI) algorithm, the multistep generalized policy iteration (MsGPI) is developed with the modification of policy evaluation. The novel algorithm not only inherits the advantages of high convergence speed and low computational complexity in the generalized PI algorithm but also further accelerates the convergence speed and reduces run time. Besides, the stability analysis, convergence analysis, and optimality analysis are given for the proposed multistep PI algorithm. Afterward, a neural network-based actor-critic structure is built for approximating the iterative control policies and value functions. Finally, a large-scale formation control problem is provided to demonstrate the performance of our developed hierarchical leader-following formation control structure and MsGPI algorithm.

DRL based low carbon economic dispatch by considering power transmission safety limitations in internet of energy

Economic dispatch, as a crucial method for ensuring the normal operation of power systems, is typically modeled as an optimization problem and solved using solvers. The introduction of low-carbon requirements has increased the complexity of the optimization problem, as economic dispatch now needs to consider effectively reducing the emission of CO2. The utilization of renewable energy can mitigate carbon emissions during power system operation, but its inherent uncertainty poses safety risks to transmission lines between microgrids and the main grid. Hence, this paper explores the issue of low-carbon economic dispatch to ensure the reliability of power transmission. We introduce a two-stage low-carbon economic dispatch model grounded in deep reinforcement learning. In the first stage, we incorporate the transmission power limits between microgrids and the main grid as constraints, generate day-ahead dispatch strategies, and determine the weights of factors in the reward function. In the second stage, we employ two methods: adjusting the weight of the penalty for exceeding the transmission limits and adding conservative safety limit constraints, to solve the problem of transmission power exceeding limits caused by uncertainties of renewable energy and prediction errors. Three deep reinforcement learning algorithms, all rooted in the Actor–Critic structure, are employed to implement the two-stage economic dispatch model. Experimental results affirm the efficacy of the proposed model in mitigating the risk of transmission power exceeding its limit, reducing carbon emissions, and minimizing operational costs.

Decentralized graph-based multi-agent reinforcement learning using reward machines

In multi-agent reinforcement learning (MARL), it is challenging for a collection of agents to learn complex temporally extended tasks. The difficulties lie in computational complexity and how to learn the high-level ideas behind reward functions. We study the graph-based Markov Decision Process (MDP), where the dynamics of neighboring agents are coupled. To learn complex temporally extended tasks, we use a reward machine (RM) to encode each agent’s task and expose reward function internal structures. RM has the capacity to describe high-level knowledge and encode non-Markovian reward functions. We propose a decentralized learning algorithm to tackle computational complexity, called decentralized graph-based reinforcement learning using reward machines (DGRM), that equips each agent with a localized policy, allowing agents to make decisions independently based on the information available to the agents. DGRM uses the actor-critic structure, and we introduce the tabular Q-function for discrete state problems. We show that the dependency of the Q-function on other agents decreases exponentially as the distance between them increases. To further improve efficiency, we also propose the deep DGRM algorithm, using deep neural networks to approximate the Q-function and policy function to solve large-scale or continuous state problems. The effectiveness of the proposed DGRM algorithm is evaluated by three case studies, two wireless communication case studies with independent and dependent reward functions, respectively, and COVID-19 pandemic mitigation. Experimental results show that local information is sufficient for DGRM and agents can accomplish complex tasks with the help of RM. DGRM improves the global accumulated reward by 119% compared to the baseline in the case of COVID-19 pandemic mitigation.

The Novel Application of Deep Reinforcement to Solve the Rebalancing Problem of Bicycle Sharing Systems with Spatiotemporal Features

Facing the Bicycle Rebalancing Problem (BRP), we established a Rebalancing Incentive System (BRIS). In BRIS, the bicycle operator proposes the method of financial compensation to encourage cylclists to detour some specific stations where the number of bikes is excessive or insufficient and access suitable sations. BRIS mainly includes two objects: the Bike Gym imitating the bicycle environment, and the Spatiotemporal Rebalancing Pricing Algorithm (STRPA) determining the amount of money which is given to the cyclist depending on time. STRPA is a deep reinforcement learning model based on the actor–critic structure, which is the core concept of this paper. In STRPA, the hierarchical principle is introduced to solve the dimensional disaster, and the graph matrix A is introduced to solve the complex node relationship. In addition, the traffic data including the bicycle have strong temporal and spatial characteristics. The gated recurrent unit (GRU), the sub-module of STRPA, can extract the temporal characteristics well, and the graph convolution network (GCN), also a sub-module of STRPA, can extract the spatial characteristic. Finally, our model is superior to the baseline model when verified on the the public bicycle data of Nanjing.

Multi-Agent Deep Reinforcement Learning Framework Strategized by Unmanned Aerial Vehicles for Multi-Vessel Full Communication Connection

In the Internet of Vessels (IoV), it is difficult for any unmanned surface vessel (USV) to work as a coordinator to establish full communication connections (FCCs) among USVs due to the lack of communication connections and the complex natural environment of the sea surface. The existing solutions do not include the employment of some infrastructure to establish USVs’ intragroup FCC while relaying data. To address this issue, considering the high-dimension continuous action space and state space of USVs, we propose a multi-agent deep reinforcement learning framework strategized by unmanned aerial vehicles (UAVs). UAVs can evaluate and navigate the multi-USV cooperation and position adjustment to establish a FCC. When ensuring FCCs, we aim to improve the IoV’s performance by maximizing the USV’s communication range and movement fairness while minimizing their energy consumption, which cannot be explicitly expressed in a closed-form equation. We transform this problem into a partially observable Markov game and design a separate actor–critic structure, in which USVs act as actors and UAVs act as critics to evaluate the actions of USVs and make decisions on their movement. An information transition in UAVs facilitates effective information collection and interaction among USVs. Simulation results demonstrate the superiority of our framework in terms of communication coverage, movement fairness, and average energy consumption, and that it can increase communication efficiency by at least 10% compared to DDPG, with the highest exceeding 120% compared to other baselines.

Intelligent Resource Allocation for Edge-Cloud Collaborative Networks: A Hybrid DDPG-D3QN Approach

Mobile edge computing (MEC) has recently emerged as an effective paradigm for computation-intensive and delay-critical applications supported by Internet of Things (IoT) devices. However, the computational resources at MEC servers are normally much smaller compared with the remote cloud server (CS). To handle the ever-increasing IoT devices and applications, the collaborative edge and cloud computing should be jointly exploited. Further, with finite block length (FBL) utilized in URLLC-supported networks, the coding error probability cannot be ignored. On these grounds, this paper investigates the dynamic offloading of FBL packets in an edge-cloud collaborative MEC system consisting of multi-mobile IoT devices (MIDs) with energy harvesting (EH), multi-edge servers, and one CS in a dynamic environment. The optimization problem is formulated to minimize the average long-term service cost defined as the weighted sum of MID energy consumption and service delay (including the uploading transmission delay, the handover cost, and the execution delay for the partial offloaded part, the local execution delay for the local processing part), with the constraints of the available resource, the energy causality, the allowable service delay, and the maximum decoding error probability. To address the problem involving both discrete and continuous variables, we propose a multi-device hybrid decision-based deep reinforcement learning (DRL) solution, named DDPG-D3QN algorithm, where the deep deterministic policy gradient (DDPG) and dueling double deep Q networks (D3QN) are invoked to tackle continuous and discrete action domains, respectively. Specifically, we improve the actor-critic structure of DDPG by combining D3QN. It utilizes the actor part of DDPG to search for the optimal offloading rate and power control of local execution. Meanwhile, it combines the critic part of DDPG with D3QN to select the optimal server for offloading. Simulation results demonstrate the proposed DDPG-D3QN algorithm has better stability and faster convergence while achieving higher rewards than the existing DRL-based methods. Furthermore, the edge-cloud collaboration has been proven to attain improved performance when compared with other offloading schemes with no collaboration between edge and cloud.

Fuzzy Control Based on Reinforcement Learning and Subsystem Error Derivatives for Strict-Feedback Systems With an Observer

In this paper, a novel optimized fuzzy adaptive control method based on tracking error derivatives of subsystems is proposed for strict-feedback systems with unmeasurable states. A cost function based on the tracking error derivative is used. It not only solves the problem that the traditional input quadratic cost function at the infinite time is unbounded, but also solves the problem that the optimal control input derived from the cost function with exponential discount factor cannot make the error asymptotically stable. Considering the case where the states are unmeasurable, a fuzzy state observer is designed which removes the restriction of the Hurwitz equation for the gain parameters. Based on reinforcement learning, the observer and error derivative cost function, an improved optimized backstepping control method is given. Using observed information and actor-critic structure to train fuzzy logic systems online, the control inputs are obtained to achieve approximate optimal control. Finally, all closed-loop signals are proved to be bounded by the Lyapunov method, and the effectiveness and advantages of the proposed algorithm are verified through two examples.

Adaptive dynamic programming for data-based optimal state regulation with experience replay

Traditional model-based control methods require accurate system dynamics. However, the dynamics are usually unknown and it is challenging to tune the control parameters manually when controlling a complex nonlinear system. In this paper, we propose a novel reinforcement learning method that combines the advantages of a model-based method, namely Adaptive Dynamic Programming (ADP), with the actor-critic method. Specifically, a linear approximate model is chosen to obtain the estimated dynamics as part of the optimal policy. Then, a designed actor-critic structure is used to obtain the sub–policy. We provide the theoretical proof of convergence and validate the proposed method through simulation experiments. The experimental results demonstrate the effectiveness of the proposed method with smaller tracking errors and faster learning speed compared with the controller trained by the actor-critic method.

Policy Optimization Adaptive Dynamic Programming for Optimal Control of Input-Affine Discrete-Time Nonlinear Systems

In this article, a policy optimization adaptive dynamic programming (POADP) method is developed for optimal control of discrete-time unknown nonlinear systems, where the iterative control policy is parameterized to optimize the iterative <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$</tex-math> </inline-formula> -function directly. The relaxed condition for the learning rate is given to guarantee the convergence of the present algorithm. Furthermore, the Polyak–Łojasiewicz inequality is introduced to analyze the optimality, i.e., the iterative <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$</tex-math> </inline-formula> -function converges to the optimum within a given computational threshold under a finite iteration, and the rate of convergence (i.e., the required minimum number of iterations) for the developed POADP method is also illustrated. To ease real implementations, the iterative <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$</tex-math> </inline-formula> -function and the iterative control policy are approximated by employing an actor–critic structure. Then, an experiment-based method is developed to obtain the initial weights of actor–critic structure. Finally, numerical simulation results of two examples are provided to validate the effectiveness of the POADP algorithm.

Adaptive Fuzzy Optimal Control for Switched Nonlinear Systems With Output Hysteresis

This paper investigates a class of switched nonlinear systems with output hysteresis, where the switching signal is arbitrary. The considered output hysteresis nonlinearity is captured by the Bouc-Wen model, which causes an uncertain gain that imposes a significant challenge on our control design. To overcome this problem, we propose an adaptive fuzzy optimal control scheme. In our scheme, the Nussbaum gain technique is used to compensate for output hysteresis, and, at each iteration, the actor-critic structure is progressively feedback and adjusted to design the optimal controller. Finally, our proposed design scheme can guarantee the stability of the nonlinear switched systems, which is demonstrated in the simulation results.

Model-Based Reinforcement Learning Control of Electrohydraulic Position Servo Systems

Even though the unprecedented success of AlphaGo Zero demonstrated reinforcement learning as a feasible complex problem solver, the research on reinforcement learning control of hydraulic systems is still void. We are motivated by the challenges presented in hydraulic systems to develop a new model-based reinforcement learning controller that achieves high-accuracy tracking at performance level and with asymptotic stability guarantees at system level. In this article, the proposed design consists of two frameworks: A recursive robust integral of the sign of the error (RISE) control approach to providing closed-loop system stability framework, and a reinforcement learning approach with actor-critic structure to dealing with the unknown dynamics or more specifically, the actor neural network is used to reduce the high feedback gain of the recursive RISE control approach by compensating the unknown dynamic while the critic neural network is integrated to improve the control performance by evaluating the filtered tracking error. A theoretical guarantee for the stability of the overall dynamic system is provided by using Lyapunov stability theory. Simulation and experimental results are provided to demonstrate improved control performance of the proposed controller.

Precise chirp control with model-based reinforcement learning for broadband frequency-swept laser of LiDAR.

Artificial intelligence (AI) has been widely used in various fields of physics and engineering in recent decades. In this work, we introduce model-based reinforcement learning (MBRL), which is an important branch of machine learning in the AI domain, to the broadband frequency-swept laser control for frequency modulated continuous wave (FMCW) light detection and ranging (LiDAR). With the concern of the direct interaction between the optical system and the MBRL agent, we establish the frequency measurement system model on the basis of the experimental data and the nonlinearity property of the system. In light of the difficulty of this challenging high-dimensional control task, we propose a twin critic network on the basis of the Actor-Critic structure to better learn the complex dynamic characteristics of the frequency-swept process. Furthermore, the proposed MBRL structure would stabilize the optimization process greatly. In the training process of the neural network, we apply a delaying strategy to the policy update and introduce a smoothing regularization strategy to the target policy to further enhance the network stability. With the well-trained control policy, the agent generates the excellent and regularly updated modulation signals to control the laser chirp precisely and an excellent detection resolution is obtained eventually. Our proposed work demonstrates that the integration of data-driven reinforcement learning (RL) and optical system control gives an opportunity to reduce the system complexity and accelerate the investigation and optimization of control systems.

Approximate Optimal Tracking Control for Partially Unknown Nonlinear Systems via an Adaptive Fixed-Time Observer

This paper investigates a novel adaptive fixed-time disturbance observer (AFXDO)-based approximate optimal tracking control architecture for nonlinear systems with partially unknown dynamic drift and perturbation under an adaptive dynamic programming (ADP) scheme. To attenuate the impact of disturbance, a novel AFXDO was designed based on the principle of a fixed-time stable system without prior information of disturbance, making disturbance observer errors converge to zero in a fixed time independent of initial estimation error. Additionally, approximate optimal control is conducted by incorporating the real-time estimation of AFXDO into a critic-only ADP framework to stabilize the dynamics of tracking errors and strike a balance between consumption and performance. In particular, to address the heavy calculation burden and oscillation phenomenon in the traditional actor–critic structure, an improved adaptive update law with a variable learning rate was developed to update the weight for adjusting the optimal cost function and optimal control policy simultaneously, avoiding the initial chattering phenomenon and achieving a prescribed convergence without resorting to dual networks. With the efforts of AFXDO and a weight law with a variable learning rate, the track errors were achieved with fast transient performance and low control consumptions in a fixed time. By revisiting Lyapunov stability, the tracking error and weight estimation error were proven to be uniformly ultimately bounded, and the designed control tended to optimal control. The simulations were carried out on quadrotor tracking to demonstrate the effectiveness of the developed control scheme, which achieves rapid convergence by lower control consumption in 4 s, where the cost function is reduced by 19.13%.