Actor-critic Network Research Articles

Adaptive finite-time optimal fuzzy control for novel constrained uncertain nonstrict feedback mixed multiagent systems via modified dynamic surface control

In this article, the finite-time stability is discussed for the novel nonlinear mixed multiagent systems (MASs) with unmodeled dynamics and constraints. Each agent is characterized as a state or output feedback system structured in the nonstrict form. The improved finite-time dynamic surface control (DSC) and the fuzzy actor-critic networks collectively constitute a novel optimal input. The fuzzy logic systems (FLSs) is introduced to approximate the unknown parts in the derivation process. By using the nonlinear transformation rules (NTRs), all the states or output are made to operate strictly within the predefined boundary conditions. The unmodeled dynamics can be solved by the constructed dynamic signals. By the aid of the compensating signals, the filtering errors can be countervailed in the DSC. The stability analysis demonstrates that all the signals are semi-globally practical finite-time stable (SGPFS). The feasibility of this scheme is explained intuitively by two simulations.

A Novel Proximal Policy Optimization Approach for Filter Design

This paper proposes a proximal policy optimization (PPO) algorithm for coupling matrix synthesis of microwave filters. With the improvement of filter design requirement, the limitations of traditional methods such as limited applicability are becoming more and more obvious. In order to improve the filter synthesis efficiency, this paper constructs a reinforcement learning algorithm based on Actor-Critic network architecture, and designs a unique filter coupling matrix synthesis reward function and action function, which can solve combinatorial optimization problems stably.

Age-of-Information based Multi-UAV Trajectories Using Deep Reinforcement Learning

The use of unmanned aerial vehicles (UAVs) as a unified platform for sensing and communication is especially relevant in environments with inadequate infrastructure. In this paper, a multi-UAV system is constructed for dynamic data collection in a resource-constrained environment. The devised approach involves the implementation of an access platform referred to as the Access UAV (A_UAV). This A_UAV orchestrates the data collection process from Inspection-UAVs (I_UAVs), each equipped with a visual sensor, facilitating the relay of collected data to the cloud. Our approach jointly considers the trajectory scheduling of A_UAV and I_UAV to collect data from specific points in a decentralized manner. Specifically, a Deep Reinforcement Learning framework utilizing an actor-critic network is formulated for A_UAV, aiming to generate an equitable access schedule for I_UAVs. Moreover, trajectory scheduling of A_UAV ensures dynamic data collection while minimizing total system energy and the Age of Information (AoI) of data arriving from I_UAVs. The simulation results validate the performance of our proposed approach against several baselines under different parameter settings.

Reinforcement learning-based adaptive tracking control for mismatched non-affine nonlinear systems with coupled uncertainties

This paper studies a reinforcement learning-based adaptive non-affine tracking control method for a class of uncertain mismatched non-affine nonlinear systems. The considered system is not only affected by external mismatched disturbances and internal uncertainties, but also influenced by the non-affine control structures. Firstly, an auxiliary integral system is developed for the purpose of isolating the non-affine control input. Secondly, by designing the actor-critic networks to evaluate the system control performance and generate the reinforcement signal, the unknown internal uncertainties can be handled. Thirdly, based on the output of reinforcement learning network, several disturbance compensation laws are constructed to address the adverse impact of matched and mismatched disturbances. As a result, a novel intelligent adaptive non-affine controller is proposed by integrating actor-critic reinforcement learning framework, disturbance compensation and adaptive laws. It has been proved that closed-loop system are stable and the tracking errors are bounded. The numerical simulation results show the effectiveness and superiority of the proposed method.

Two-Stage Evolutionary Reinforcement Learning for Enhancing Exploration and Exploitation

The integration of Evolutionary Algorithm (EA) and Reinforcement Learning (RL) has emerged as a promising approach for tackling some challenges in RL, such as sparse rewards, lack of exploration, and brittle convergence properties. However, existing methods often employ actor networks as individuals of EA, which may constrain their exploratory capabilities, as the entire actor population will stop evolution when the critic network in RL falls into local optimal. To alleviate this issue, this paper introduces a Two-stage Evolutionary Reinforcement Learning (TERL) framework that maintains a population containing both actor and critic networks. TERL divides the learning process into two stages. In the initial stage, individuals independently learn actor-critic networks, which are optimized alternatively by RL and Particle Swarm Optimization (PSO). This dual optimization fosters greater exploration, curbing susceptibility to local optima. Shared information from a common replay buffer and PSO algorithm substantially mitigates the computational load of training multiple agents. In the subsequent stage, TERL shifts to a refined exploitation phase. Here, only the best individual undergoes further refinement, while the rest individuals continue PSO-based optimization. This allocates more computational resources to the best individual for yielding superior performance. Empirical assessments, conducted across a range of continuous control problems, validate the efficacy of the proposed TERL paradigm.

Learning strategies for underwater robot autonomous manipulation control

Autonomous manipulation operations represent the high intelligent coordination from robotic vision and control, it is also a symbol of the advances of robotic intelligence. The limitations of visual sensing and the increasingly complex experimental conditions make autonomous manipulation operations more difficult, particularly for deep reinforcement learning methods, which can enhance robotic control intelligence but require a lot of training process. Due to the high-dimensional continuous state space and continuous action space characteristics of underwater operations, this paper adopts a policy-based reinforcement learning method as the foundational approach. To address the issues of instability and low convergence efficiency in traditional policy-based reinforcement learning algorithms during the learning process, this paper proposes a novel policy learning method. This method adopts the Proximal Policy Optimization algorithm (PPOClip) and optimizes it through an actor-critic network. The aim is to improve the stability and effectiveness of convergence in the learning process. In the underwater training environment, a new reward shaping scheme has been designed to address the issue of reward sparsity during the training process. The manually crafted dense reward function is utilized as attractive and repulsive potential functions for goal manipulation and obstacle avoidance. On the highly complex underwater manipulation and training environment, transferred learning algorithm has been established to reduce the training times and compensate the differences between the simulation and experiment. Simulations and tank experiments have verified the performance of the proposed strategy learning method.

Reinforcement learning-based adaptive spiral-diving Manoeuver guidance method for reentry vehicles subject to unknown disturbances

Abstract This paper proposed a reinforcement learning-based adaptive guidance method for a class of spiral-diving manoeuver guidance problems of reentry vehicles subject to unknown disturbances. First, the desired proportional navigation guidance law is designed for the vehicle based on the initial conditions, terminal constraints and the curve involute principle. Then, the first-order multivariable nonlinear guidance command tracking model considering unknown disturbances is established. And the controller design problem caused by the coupling of control variables is overcome by introducing the coordinate transformation technique. Moreover, the actor-critic networks and corresponding adaptive weight update laws are designed to cope with unknown disturbances. With the help of Lyapunov direct method, the stability of the system is proved. Subsequently, the range values of the guidance parameters are analysed. Finally, the validity as well as superiority of the proposed method are verified by numerical simulations.

An Adaptive Contention Window using Actor-Critic Reinforcement Learning Algorithm for Vehicular Ad-hoc NETworks

The MAC layer IEEE 802.11p standard’s collision avoidance mechanism is sub-optimal due to the use of a Binary Exponential Backoff (BEB). This technique has an increase in the backoff slot time to minimize the probability of subsequent collisions after the occurrence of a collision. This leads to a degradation of radio bandwidth wastage, especially when managing access to a dense number of stations. An incorrect backoff setting can also increase the likelihood of collisions and decrease network performance. For adaptive CW selection, the proposed model is based on a Reinforcement Learning (RL) Actor-Critic (AC) network to adapt the CW value by learning the VANET environment. NS-3 for VANET is utilized to test and compute the optimal CW values. The proposed AC and EAC outperform traditional BEB used in the IEEE 802.11p standard as the number of vehicles increases. The throughput is enhanced by 54% and 69% respectively for AC and Enhanced AC (EAC). Similarly, the end-to-end delay for AC is reduced by 19%, and for EAC it’s reduced by 25%.

IACPPO: A deep reinforcement learning-based model for warehouse inventory replenishment

Inventory cost is a significant factor in Supply Chain Management (SCM), and an effective replenishment strategy can reduce warehouse operation costs. However, traditional replenishment strategies often struggle to meet the complex and ever-changing demands of real-world warehouse scenarios. Moreover, the spatiotemporal heterogeneity of commodity demand and inventory cost poses significant challenges to time series prediction models, as individual training strategies for different commodities significantly increase modeling and time costs. To address these issues, we propose a replenishment model called IACPPO, which incorporates the Advantage Actor-Critic (A2C) algorithm with the Proximal Policy Optimization (PPO) algorithm. Firstly, we introduce gated recurrent unit (GRU) and Attention Mechanisms into the Actor-Critic network to analyze data state spaces for probabilistic modeling and extracting valid information from environmental state sequences by memory reasoning and focusing on critical state sequences; additionally, we fuse the A2C algorithm with the PPO algorithm to train the whole network simultaneously to obtain the replenishment strategy. Finally, experimental results on two different real-world inventory datasets show that using the IACPPO model has achieved the best cost control strategies in most experimental validations of replenishment strategies.

An Adaptive Metadata Management Scheme Based on Deep Reinforcement Learning for Large-Scale Distributed File Systems

A major challenge confronting today’s distributed metadata management schemes is how to meet the dynamic requirements of various applications through effectively mapping and migrating metadata nodes to different metadata servers (MDS’s). Most of the existing works dynamically reallocate nodes to different servers adopting history-based coarse-grained methods, failing to make a timely and efficient update on the distribution of nodes. In this paper, we present the first deep reinforcement learning-leveraged distributed metadata management scheme, AdaM, to address the aforementioned dilemma. AdaM is an adaptive fine-grained metadata management scheme that trains an actor-critic network to migrate “hot” metadata nodes to different MDS’s based on its observations of the current “states” (i.e., access pattern, the structure of namespace tree and current distribution of nodes on MDS’s). Adaptive to varying access patterns, AdaM can automatically migrate hot metadata nodes among servers to keep load balancing while maintaining metadata locality. Besides, we propose a self-adaptive metadata cache policy, which dynamically combines the two strategies of managing caches on the server side and the client side to gain better query performance. Last but not least, we design a distributed metadata processing 2PC Protocol called MST-based 2PC to ensure data consistency. Experiments on a real-world dataset demonstrate the superiority of our proposed method over other schemes.

EdgeAISim: A toolkit for simulation and modelling of AI models in edge computing environments

To meet next-generation Internet of Things (IoT) application demands, edge computing moves processing power and storage closer to the network edge to minimize latency and bandwidth utilization. Edge computing is becoming increasingly popular as a result of these benefits, but it comes with challenges such as managing resources efficiently. Researchers are utilising Artificial Intelligence (AI) models to solve the challenge of resource management in edge computing systems. However, existing simulation tools are only concerned with typical resource management policies, not the adoption and implementation of AI models for resource management, especially. Consequently, researchers continue to face significant challenges, making it hard and time-consuming to use AI models when designing novel resource management policies for edge computing with existing simulation tools. To overcome these issues, we propose a lightweight Python-based toolkit called EdgeAISim for the simulation and modelling of AI models for designing resource management policies in edge computing environments. In EdgeAISim, we extended the basic components of the EdgeSimPy framework and developed new AI-based simulation models for task scheduling, energy management, service migration, network flow scheduling, and mobility support for edge computing environments. In EdgeAISim, we have utilized advanced AI models such as Multi-Armed Bandit with Upper Confidence Bound, Deep Q-Networks, Deep Q-Networks with Graphical Neural Network, and Actor-Critic Network to optimize power usage while efficiently managing task migration within the edge computing environment. The performance of these proposed models of EdgeAISim is compared with the baseline, which uses a worst-fit algorithm-based resource management policy in different settings. Experimental results indicate that EdgeAISim exhibits a substantial reduction in power consumption, highlighting the compelling success of power optimization strategies in EdgeAISim. The development of EdgeAISim represents a promising step towards sustainable edge computing, providing eco-friendly and energy-efficient solutions that facilitate efficient task management in edge environments for different large-scale scenarios.

Mean Field Multi-Agent Reinforcement Learning Method for Area Traffic Signal Control

Reinforcement learning is an effective method for adaptive traffic signal control in urban transportation networks. As the number of training rounds increases, the optimal control strategy is learned, and the learning capabilities of deep neural networks are further enhanced, thereby avoiding the limitations of traditional signal control methods. However, when faced with the sequential decision tasks of regional signal control, it encounters issues such as the curse of dimensionality and environmental non-stationarity. To address the limitations of traditional reinforcement learning algorithms applied to multiple intersections, the mean field theory is applied. This models the traffic signal control problem at multiple intersections within a region as interactions between individual intersections and the average effects of neighboring intersections. By decomposing the Q-function through bilateral estimation between the agent and its neighbors, this method reduces the complexity of interactions between agents while preserving global interactions between the agents. A traffic signal control model based on Mean Field Multi-Agent Reinforcement Learning (MFMARL) was constructed, containing two algorithms: Mean Field Q-Network Area Traffic Signal Control (MFQ-ATSC) and Mean Field Actor-Critic Network Area Traffic Signal Control (MFAC-ATSC). The model was validated using the SUMO simulation platform. The experimental results indicate that across different metrics, such as average speed, the mean field reinforcement learning method outperforms classical signal control methods and several existing approaches.

Quantum Multiagent Actor–Critic Networks for Cooperative Mobile Access in Multi-UAV Systems

This paper proposes a novel algorithm, named quantum multi-agent actor-critic networks (QMACN) for autonomously constructing a robust mobile access system employing multiple unmanned aerial vehicles (UAVs). In the context of facilitating collaboration among multiple unmanned aerial vehicles (UAVs), the application of multi-agent reinforcement learning (MARL) techniques is regarded as a promising approach. These methods enable UAVs to learn collectively, optimizing their actions within a shared environment, ultimately leading to more efficient cooperative behavior. Furthermore, the principles of a quantum computing (QC) are employed in our study to enhance the training process and inference capabilities of the UAVs involved. By leveraging the unique computational advantages of quantum computing, our approach aims to boost the overall effectiveness of the UAV system. However, employing a QC introduces scalability challenges due to the near intermediate-scale quantum (NISQ) limitation associated with qubit usage. The proposed algorithm addresses this issue by implementing a quantum centralized critic, effectively mitigating the constraints imposed by NISQ limitations. Additionally, the advantages of the QMACN with performance improvements in terms of training speed and wireless service quality are verified via various data-intensive evaluations. Furthermore, this paper validates that a noise injection scheme can be used for handling environmental uncertainties in order to realize robust mobile access.

Cross-Region Courier Displacement for On-Demand Delivery With Multi-Agent Reinforcement Learning

On-demand delivery has become prevailing for people to order meals and groceries online, especially during the pandemic. It is essential to dispatch massive orders to limited couriers to satisfy on-demand delivery users, especially during peak hours. Existing studies mainly focus on order dispatching within a region, and they are challenging to be applied to the cross-region courier displacement problem due to (i) unique practical factors, including regional spatial-temporal demand-supply dynamics and strict delivery time constraints, and (ii) the large-scale setting and high-dimensional decision space given massive couriers in on-demand delivery. To address these challenges, in this work, we propose an efficient cross-region courier displacement framework, i.e., Courier Displacement Reinforcement Learning (short for CDRL) based on centralized multi-agent actor-critic, which first design the actor-critic network with a time-varying displacement intensity control module to capture demand-supply dynamics and utilize the centralized training and decentralized execution multi-agent framework to address the large-scale coordination. One-month real-world order records collected from one of the biggest on-demand delivery services in the world are utilized to show the performance of our design. The extensive results show that our method offers a 47.97% of increase in balancing supply and demand and reduces idle ride time by 24.62% simultaneously.

Generative Adversarial Reward Learning for Generalized Behavior Tendency Inference

Recent advances in reinforcement learning have inspired increasing interest in learning user modeling adaptively through dynamic interactions, e.g., in reinforcement learning based recommender systems. In most reinforcement learning applications, reward functions provide the critical guideline for optimization. However, current reinforcement learning-based methods rely on manually-defined reward functions, which cannot adapt to dynamic, noisy environments. Moreover, they generally use task-specific reward functions that sacrifice generalization ability. We propose a generative inverse reinforcement learning for user behavioral preference modeling to address the above issues. Instead of using predefined reward functions, our model can automatically learn the rewards from user's actions based on discriminative actor-critic network and Wasserstein GAN. Our model provides a general approach to characterizing and explaining underlying behavioral tendencies. Our experiments show our method outperforms state-of-the-art methods in several scenarios, namely traffic signal control, online recommender systems, and scanpath prediction.

Deep reinforcement learning for dynamic flexible job shop scheduling problem considering variable processing times

In recent years, the uncertainties and complexity in the production process, due to the boosted customized requirements, has dramatically increased the difficulties of Dynamic Flexible Job Shop Scheduling (DFJSP). This paper investigates a new DFJSP model taking into account the minimum completion time under the condition of machine processing time uncertainty, e.t. VPT-FJSP problem. In the formulated VPT-FJSP process, each workpiece needs to be processed by required machine at a certain time slot where Markov decision process (MDP) and reinforcement learning methods are adopted to solve VPT-FJSP. The agent designed in this paper employs the Proximal Policy Optimization(PPO) algorithm in deep reinforcement learning, which includes the Actor-Critic network. The input of the network is to extract the processing information matrix and to embed some advanced states in the workshop by graph neural network, which enables the agent to learn the complete state of the environment. Finally, we train and test the proposed framework on the canonical FJSP benchmark, and the experimental results show that our framework can make agent better than genetic algorithm and ant colony optimization in most cases, 94.29% of static scheduling. It is also shown superiority compared to the scheduling rules in dynamic environment and has demonstrated strong robustness in solving VPT-FJSP. Furthermore, this study conducted tests to assess the generalization capability of the agent on VPT-FJSP at different scales. In terms of exploring Makespan minimization, the agent outperformed four priority scheduling rules. These results indicate that the proposed dynamic scheduling framework and PPO algorithm are more effective in achieving superior solutions.

A Deep Reinforcement Learning-Based Path-Following Control Scheme for an Uncertain Under-Actuated Autonomous Marine Vehicle

In this article, a deep reinforcement learning-based path-following control scheme is established for an under-actuated autonomous marine vehicle (AMV) in the presence of model uncertainties and unknown marine environment disturbances is presented. By virtue of light-of-sight guidance, a surge-heading joint guidance method is developed within the kinematic level, thereby enabling the AMV to follow the desired path accurately. Within the dynamic level, model uncertainties and time-varying environment disturbances are taken into account, and the reinforcement learning control method using the twin-delay deep deterministic policy gradient (TD3) is developed for the under-actuated vehicle, where path-following actions are generated via the state space and hybrid rewards. Additionally, actor-critic networks are developed using the long-short time memory (LSTM) network, and the vehicle can successfully make a decision by the aid of historical states, thus enhancing the convergence rate of dynamic controllers. Simulation results and comprehensive comparisons on a prototype AMV demonstrate the remarkable effectiveness and superiority of the proposed LSTM-TD3-based path-following control scheme.

Model-free algorithm for consensus of discrete-time multi-agent systems using reinforcement learning method

In this work, we investigate consensus issues of discrete-time (DT) multi-agent systems (MASs) with completely unknown dynamic by using reinforcement learning (RL) technique. Different from policy iteration (PI) based algorithms that require admissible initial control policies, this work proposes a value iteration (VI) based model-free algorithm for consensus of DTMASs with optimal performance and no requirement of admissible initial control policy. Firstly, in order to utilize RL method, the consensus problem is modeled as an optimal control problem of tracking error system for each agent. Then, we introduce a VI algorithm for consensus of DTMASs and give a novel convergence analysis for this algorithm, which does not require admissible initial control input. To implement the proposed VI algorithm to achieve consensus of DTMASs without information of dynamics, we construct actor-critic networks to online estimate the value functions and optimal control inputs in real time. At last, we give some simulation results to show the validity of the proposed algorithm.

Intermittent Event-Triggered Optimal Leader-Following Consensus for Nonlinear Multi-Agent Systems Via Actor-Critic Algorithm.

This article investigates the intermittent event-triggered optimal leader-following consensus for nonlinear multi-agent systems (MASs) utilizing the actor-critic algorithm. First, we propose a novel distributed intermittent event-triggered control strategy, and a sufficient criterion is obtained to guarantee the leader-following consensus of MASs by establishing a novel piecewise differential inequality. Next, the intermittent event-triggered optimal control strategy is delicately given. Remarkably, the optimality of MASs is proven based on policy iteration and the convergence of the closed-loop system is also proved based on the Lyapunov stability theory. Then, the intermittent event-triggered approximate optimal control strategy is designed via an actor-critic network whose weights are only updated at the trigger instants. Furthermore, the Zeno behavior can be excluded in this article. Finally, two simulation examples further verify the effectiveness of the proposed scheme.

Scalable Deep Reinforcement Learning-Based Online Routing for Multi-Type Service Requirements

Emerging applications raise critical QoS requirements for the Internet. The improvements in flow classification technologies, software-defined networks (SDN), and programmable network devices make it possible to fast identify users' requirements and control the routing for fine-grained traffic flows. Meanwhile, the problem of optimizing the forwarding paths for traffic flows with multiple QoS requirements in an online fashion is not addressed sufficiently. To address the problem, we propose DRL-OR-S, a highly scalable online routing algorithm using multi-agent deep reinforcement learning. DRL-OR-S adopts a comprehensive reward function, an efficient learning algorithm, and a novel deep neural network structure to learn appropriate routing strategies for different types of flow requirements. In order to enhance the generalization and scalability, we propose a novel graph-based actor-critic network architecture and a carefully designed input state for DRL-OR-S. To accelerate the training process and guarantee reliability, we further introduce an NN-simulator for efficient offline training and a safe learning mechanism to avoid unsafe routes during the online routing process. We implement DRL-OR-S under SDN architecture and conduct Mininet-based experiments using real network topologies and traffic traces. The results validate that DRL-OR-S can well satisfy the requirements of latency-sensitive, throughput-sensitive, latency-throughput-sensitive, and latency-loss-sensitive flows at the same time, while exhibiting great adaptiveness and reliability under the scenarios of link failure, traffic change, unseen large topology and partial deployment.