Actor Critic Research Articles

Deep Reinforcement Learning-Based Diving/Pull-out Control for Bioinspired Morphing UAVs

Next-generation unmanned aerial vehicles (UAVs) should possess the capability to autonomously perform multiple tasks with agility and efficiency in complex obstructive environments and over open terrains. Morphing UAVs can autonomously transform in response to changes in flight environments and tasks, always maintaining an optimal aerodynamic profile. The falcon-inspired morphing UAV folds/twists its wings/tail simultaneously to accomplish diving/pull-out flight during the predation process. In the diving/pull-out flight, falcon-inspired morphing UAVs are able to balance maneuverability and stability, which is hard to realize by current control methods. This paper proposes a deep reinforcement learning (DRL)-based diving/pull-out cooperative control strategy. Considering the continuity of the state space and the action space of morphing UAVs, the deep deterministic policy gradient (DDPG) algorithm based on the actor-critic (AC) network is adopted and refined. With the aim of ensuring the smoothness of the flight action, the proposed DRL-based strategy is tasked with controlling multiple data frames of airspeed, altitude and pitch angle to desired reference values. Numerical experiments have been conducted on fixed-speed ascent/descent flight and diving/pull-out maneuvers flight missions. The results demonstrate the superiority of the proposed DRL-based control strategy compared with a classical proportional-integral-derivative (PID) control strategy. Furthermore, the proposed DRL controller is shown to generalize well to the random white noises which are added to gyroscope measurements and wind disturbance in flight.

Deep reinforcement learning with a critic-value-based branch tree for the inverse design of two-dimensional optical devices

In optical engineering, designing a device or a system with the desired property is an important but challenging task. It is relatively straightforward to compute the physical properties of a given design; however, there is no general method for the reverse, i.e., designing with desired properties. To address this problem with a computational method, this paper proposes a deep reinforcement learning-based inverse design framework consisting of two methods: Inverse DEsign Agent (IDEA) and Critic-Value-based Branch Tree (CVBT) algorithm. IDEA is a deep reinforcement learning model based on the Advantage Actor–Critic (A2C) method using a deep learning simulator as a replacement for a real environment, significantly reducing training time compared to conventional methods. The CVBT algorithm suggests several design candidates using the critic values of IDEA, while conventional methods propose only one candidate. In this study, the proposed framework was applied to a two-dimensional optical device design problem. Experimental results with untrained target properties demonstrated that the proposed model, IDEA-CVBT, achieved state-of-the-art performance in terms of accuracy and stability. For instance, in a scenario with a binary type design, IDEA-CVBT exhibited an accuracy of 91.5%, while conventional deep reinforcement models showed 36.1% and 7.4% accuracies. Extensive analyses verified that IDEA-CVBT can be employed as an assistance system for engineers since IDEA-CVBT suggests several appropriate candidates that satisfy target properties.

Design of Travel Route Identification and Scheduling System Based on Artificial Intelligence-Aided Image Segmentation.

This study designs a travel recognition and scheduling system using artificial intelligence and image segmentation techniques. To address the problem of low division quality of current point division algorithms, this study proposes a streaming graph division model based on a sliding window (GraphWin), which dynamically adjusts the amount of information (vertex degree information and adjacency information) referenced at each division according to the current division quality and division time by introducing a sliding window mechanism, to achieve the highest possible division while allowing loss of certain division efficiency. The goal is to improve the division quality as much as possible while allowing a certain loss of division efficiency. To meet the user's need to travel through multiple destinations with the shortest route, this thesis proposes a deep reinforcement learning actor-critic (AC)-based multiobjective point path planning algorithm. The algorithm builds a strategy network and an evaluation network based on actor-critic's multiobjective point path planning, updates the strategy network and evaluation network parameters using AC optimization training, reduces the reliance of the algorithm model on a large amount of high-quality label data, and speeds up the convergence speed of the deep reinforcement learning algorithm by pretraining, finally completing the multiobjective point access sequential path planning task. Finally, the personalized travel recommendation system is designed and implemented, and the system performance analysis is conducted to clarify the system requirements in terms of functional and nonfunctional aspects: the system architecture, system functional modules, and database tables are designed to conduct use case testing of the main functional modules of the system, and the usability of the attraction recommendation algorithm is verified through the concrete implementation of the functional modules such as attraction recommendation in the system.

Proximal Policy Optimization With Policy Feedback

Proximal policy optimization (PPO) is a deep reinforcement learning algorithm based on the actor–critic (AC) architecture. In the classic AC architecture, the Critic (value) network is used to estimate the value function while the Actor (policy) network optimizes the policy according to the estimated value function. The efficiency of the classic AC architecture is limited due that the policy does not directly participate in the value function update. The classic AC architecture will make the value function estimation inaccurate, which will affect the performance of the PPO algorithm. For improvement, we designed a novel AC architecture with policy feedback (AC-PF) by introducing the policy into the update process of the value function and further proposed the PPO with policy feedback (PPO-PF). For the AC-PF architecture, the policy-based expected (PBE) value function and discount reward formulas are designed by drawing inspiration from expected Sarsa. In order to enhance the sensitivity of the value function to the change of policy and to improve the accuracy of PBE value estimation at the early learning stage, we proposed a policy update method based on the clipped discount factor. Moreover, we specifically defined the loss functions of the policy network and value network to ensure that the policy update of PPO-PF satisfies the unbiased estimation of the trust region. Experiments on Atari games and control tasks show that compared to PPO, PPO-PF has faster convergence speed, higher reward, and smaller variance of reward.

Soft Actor–Critic Algorithm Featured Residential Demand Response Strategic Bidding for Load Aggregators

Load aggregators (LAs) gather regional demand response (DR) resources as a representative of regional users to transact with the grid. They achieve this by facilitating residential customers to flexibly participate in DR programs. This article proposes a residential DR strategic bidding approach for LAs to improve system reliability while maximizing their financial advantages. It is based on the deep reinforcement learning (DRL) algorithm and is capable of overcoming the computational limitations of conventional model-based strategy design. A probabilistic model considering the electricity usage habits and response willingness was developed to reliably quantify the uncertainty responsive behaviors. A DR bidding model considering bilateral interaction with declaration risk was developed to balance the contradictory relationship between subsidy prices with uncertain DR quantities. In addition, the soft actor-critic algorithm was implemented to optimize the decision-making process, and thereby, optimize the continuous-action-space problem of maximizing the profits of LAs by DR scheduling. The proposed approach demonstrated its effectiveness and superiority in diverse DR situations, compared with a conventional method and another DRL approach.

A Proposed Priority Pushing and Grasping Strategy Based on an Improved Actor-Critic Algorithm

The most basic and primary skills of a robot are pushing and grasping. In cluttered scenes, push to make room for arms and fingers to grasp objects. We propose a modified Actor-Critic (A-C) framework for deep reinforcement learning, Cross-entropy Softmax A-C (CSAC), and use the Prioritized Experience Replay (PER) based on the theoretical foundation and main methods of deep reinforcement learning, combining the advantages of algorithms based on value functions and policy gradients. The grasping model is trained using self-supervised learning to achieve end-to-end mapping from image to propulsion and grasping action. A vision module and an action module have been created out of the entire algorithm framework. The prioritized experience replay is improved to further improve the CSAC-PER algorithm for model sample diversity and robot exploration performance during robot grasping training. The experience replay buffer is dynamically sampled using the prior beta distribution and the dynamic sampling algorithm based on the beta distribution (CSAC-β) is proposed based on the CSAC algorithm. Despite its low initial efficiency, the experimental simulation results show that the CSAC-β algorithm eventually achieves good results and has a higher grasping success rate (90%).

Action-Based Load Balancing Technique in Cloud Network Using Actor-Critic-Swarm Optimization

Increasing scale of task in cloud network leads to problem in load balancing and its improvement in parameters. In this paper, we proposed a hybrid scheduling policy which is hybrid of both Particle Swarm Optimization (PSO) algorithm and actor-critic algorithm named as Hybrid Particle Swarm Optimization Actor Critic (HPSOAC) to solve this issue. This hybrid scheduling policy helps to each agent to improve an individual learning as well as learning through exchanging information among other agents. An experiment is carried out by the help of Python simulator with TensorFlow. Outcome shows that our proposed scheduling policy reduces 5.16% and 10.86% in energy consumption, reduces 7.13% and 10.04% in makespan time, and has marginally better resource utilization over Deep Q-network (DQN) and Q-learning based on Modified Particle Swarm Optimization (QMPSO) algorithm, respectively.

Comparison of On-Policy Deep Reinforcement Learning A2C with Off-Policy DQN in Irrigation Optimization: A Case Study at a Site in Portugal

Precision irrigation and optimization of water use have become essential factors in agriculture because water is critical for crop growth. The proper management of an irrigation system should enable the farmer to use water efficiently to increase productivity, reduce production costs, and maximize the return on investment. Efficient water application techniques are essential prerequisites for sustainable agricultural development based on the conservation of water resources and preservation of the environment. In a previous work, an off-policy deep reinforcement learning model, Deep Q-Network, was implemented to optimize irrigation. The performance of the model was tested for tomato crop at a site in Portugal. In this paper, an on-policy model, Advantage Actor–Critic, is implemented to compare irrigation scheduling with Deep Q-Network for the same tomato crop. The results show that the on-policy model Advantage Actor–Critic reduced water consumption by 20% compared to Deep Q-Network with a slight change in the net reward. These models can be developed to be applied to other cultures with high production in Portugal, such as fruit, cereals, and wine, which also have large water requirements.

Safety-constrained reinforcement learning with a distributional safety critic

Safety is critical to broadening the real-world use of reinforcement learning. Modeling the safety aspects using a safety-cost signal separate from the reward and bounding the expected safety-cost is becoming standard practice, since it avoids the problem of finding a good balance between safety and performance. However, it can be risky to set constraints only on the expectation neglecting the tail of the distribution, which might have prohibitively large values. In this paper, we propose a method called Worst-Case Soft Actor Critic for safe RL that approximates the distribution of accumulated safety-costs to achieve risk control. More specifically, a certain level of conditional Value-at-Risk from the distribution is regarded as a safety constraint, which guides the change of adaptive safety weights to achieve a trade-off between reward and safety. As a result, we can compute policies whose worst-case performance satisfies the constraints. We investigate two ways to estimate the safety-cost distribution, namely a Gaussian approximation and a quantile regression algorithm. On the one hand, the Gaussian approximation is simple and easy to implement, but may underestimate the safety cost, on the other hand, the quantile regression leads to a more conservative behavior. The empirical analysis shows that the quantile regression method achieves excellent results in complex safety-constrained environments, showing good risk control.

Fixed-time convergence attitude control for a tilt trirotor unmanned aerial vehicle based on reinforcement learning

This paper presents a new nonlinear robust attitude control strategy for the tilt trirotor unmanned aerial vehicle (UAV). Fixed-time convergence control of the UAV’s attitude tracking errors under the effects of model uncertainties and unknown external disturbances is achieved by utilizing the proposed control design. Actor–critic (AC) structure based neural networks are trained only with the information of the UAV’s inputs and outputs data, to handle the UAV’s modeling uncertainties with bounded estimation error. Then a sliding-mode based fixed-time controller is designed to compensate the approximation error of the neural networks and the unknown external disturbances. Based on the Lyapunov stability theory, the stability analysis of the closed-loop system is presented. The performance of the presented nonlinear robust control strategy is validated through the real-time flight experiments.

Reinforcement Learning based cooperative longitudinal control for reducing traffic oscillations and improving platoon stability

Stop-and-go traffic poses significant challenges to the efficiency and safety of traffic operations. In this study, a cooperative longitudinal control based on Soft Actor Critic (SAC) Reinforcement Learning (RL) is proposed to address this issue. The reward function is carefully designed to consider vehicle cooperation and to achieve three main objectives: safety, efficiency, and oscillation dampening. A global performance metric for oscillation dampening is proposed to evaluate the developed RL and other baseline models. Depending on the number of preceding vehicles that can share maneuver information, two models RL-1 and RL-2 are proposed and compared with human driven (HD) and an adaptive cruise control (ACC) model using the HighD and simulated data. It is found that with information from additional preceding vehicles, RL-2 can dampen shockwaves more efficiently. Specifically, RL-1 and RL-2 decrease traffic oscillation by 15%-36% and 15%-42%, respectively, while HD amplifies the oscillation by 14–37%. The ACC model can also dampen shockwaves but is not as effective as RL-1 and RL-2. The two RL control methods are further evaluated based on data collected using a commercial Model X vehicle. Compared with the commercial Model X ACC vehicle in some controlled settings, the proposed RL methods can better dampen the stop-and-go waves by generating smaller oscillation growth, overshooting, and average acceleration/deceleration rate change, suggesting that they can generalize well in a new but similar environment. Finally, the RL methods are evaluated considering a platoon of vehicles with different RL penetration rates. The results show that they consistently outperform HD and ACC in dampening shockwaves.

The Intelligent Path Planning System of Agricultural Robot via Reinforcement Learning.

Agricultural robots are one of the important means to promote agricultural modernization and improve agricultural efficiency. With the development of artificial intelligence technology and the maturity of Internet of Things (IoT) technology, people put forward higher requirements for the intelligence of robots. Agricultural robots must have intelligent control functions in agricultural scenarios and be able to autonomously decide paths to complete agricultural tasks. In response to this requirement, this paper proposes a Residual-like Soft Actor Critic (R-SAC) algorithm for agricultural scenarios to realize safe obstacle avoidance and intelligent path planning of robots. In addition, in order to alleviate the time-consuming problem of exploration process of reinforcement learning, this paper proposes an offline expert experience pre-training method, which improves the training efficiency of reinforcement learning. Moreover, this paper optimizes the reward mechanism of the algorithm by using multi-step TD-error, which solves the probable dilemma during training. Experiments verify that our proposed method has stable performance in both static and dynamic obstacle environments, and is superior to other reinforcement learning algorithms. It is a stable and efficient path planning method and has visible application potential in agricultural robots.

Graph neural network and multi-agent reinforcement learning for machine-process-system integrated control to optimize production yield

In this paper, an integrated control framework is proposed for the optimization of the production yield by integrating different levels of a manufacturing system, including system, process, and machine levels. The manufacturing system is modeled as a graph by treating machines as nodes and material flows as links. The graph model enjoys high flexibility and is able to incorporate all relevant real-time information across all levels of the manufacturing system in the dynamic node feature. Since the real-time tool state is essential for decision making, Recursive Bayesian Estimation (RBE) is adopted to reduce the tool state observations through sensors and machine learning models and provide more accurate tool state estimation to be included into the graph node feature. With the graph model, Graph Neural Network (GNN) is applied to process the node features to generate node embedding that reflects both local and global information. For the integrated control purpose, each machine node is then be treated as a distributed agent in Multi-Agent Reinforcement Learning (MARL) that conditions its policy on the node embedding from GNN. State-of-the-art GNN and MARL algorithms, namely Graph Attention Network (GAT) and Value Decomposition Actor Critic (VDAC), are implemented to train learnable parameters in GNN-MARL networks to learn the optimal multi-agent policy. Extensive numerical experiments and analysis proves the effectiveness of the proposed integrated control framework.

Continuous-Time Convergence Rates in Potential and Monotone Games

In this paper, we provide exponential rates of convergence to the interior Nash equilibrium for continuous-time dual-space game dynamics such as mirror descent (MD) and actor-critic (AC). We perform our analysis in $N$-player continuous concave games that satisfy certain monotonicity assumptions while possibly also admitting potential functions. In the first part of this paper, we provide a novel relative characterization of monotone games and show that MD and its discounted version converge with $\mathcal{O}(e^{-\beta t})$ in relatively strongly and relatively hypomonotone games, respectively. In the second part of this paper, we specialize our results to games that admit a relatively strongly concave potential and show that AC converges with $\mathcal{O}(e^{-\beta t})$. These rates extend their known convergence conditions. Simulations are performed which empirically back up our results.

Proactive Content Caching Based on Actor–Critic Reinforcement Learning for Mobile Edge Networks

Mobile edge caching/computing (MEC) has emerged as a promising approach for addressing the drastic increasing mobile data traffic by bringing high caching and computing capabilities to the edge of networks. Under MEC architecture, content providers (CPs) are allowed to lease some virtual machines (VMs) at MEC servers to proactively cache popular contents for improving users’ quality of experience. The scalable cache resource model rises the challenge for determining the ideal number of leased VMs for CPs to obtain the minimum expected downloading delay of users at the lowest caching cost. To address these challenges, in this paper, we propose an actor-critic (AC) reinforcement learning based proactive caching policy for mobile edge networks without the prior knowledge of users’ content demand. Specifically, we formulate the proactive caching problem under dynamical users’ content demand as a Markov decision process and propose a AC based caching algorithm to minimize the caching cost and the expected downloading delay. Particularly, to reduce the computational complexity, a branching neural network is employed to approximate the policy function in the actor part. Numerical results show that the proposed caching algorithm can significantly reduce the total cost and the average downloading delay when compared with other popular algorithms.

Parallel Deep Reinforcement Learning Method for Gait Control of Biped Robot

In this brief, a parallel Deep Deterministic Policy Gradient (DDPG) algorithm is presented for biped robot gait control. Biped robot gait control is a high-dimensional continuous problem. It is challenging to obtain a fast and stable gait. Traditional methods cannot fully utilize autonomous exploration capability of a biped robot. A multiple Actor-Critic (AC) network is established to expand the scope of exploration and improve training efficiency. For optimizing experience replay mechanism, an experience filtering unit is introduced, and a cosine similarity method is used to classify experience. Then, a Markov Decision Process (MDP) model based on knowledge and experience is designed to solve the problem of sparse rewards. Finally, experimental results show that the parallel DDPG algorithm can make the biped robot walk more quickly and stably, and the speed reaches 0.62 m/s.

Power system fault restoration based on A2C with the consideration of renewables

Although the power system has made great progress in security and stability, it is still difficult to completely avoid the occurrence of large-scale blackouts caused by various accidental factors under the background of high proportion of renewables. In order to improve the rapidity and reliability of power system fault restoration, this paper proposes a power system fault restoration strategy based on the Advantage Actor Critic. Under the premise of ensuring the priority restoration of loads with high demand on the user side, the overall objective function is to take the least total power loss load, the smallest network loss, the smallest restoration cost and the least number of switching operations into consideration. Test results on a modified IEEE39 system have verified the superiority of the model and recovery strategy.

Imitation Learning Based Heavy-Hitter Scheduling Scheme in Software-Defined Industrial Networks

To realize flexible networking and on-demand topology reconstructing, Software-Defined Industrial Networks(SDIN) are increasingly embracing the flat structure. Similar to SDN, SDIN suffers from low traffic scheduling efficiency caused by large and imbalanced flows, known as the heavy hitters problem. Due to such heavy hitters, industrial networks may fail to satisfy application's QoS requirements, which results in more severe damages. To improve flow scheduling efficiency under heavy hitters, this article introduces a novel Imitation Learning-based Flow Scheduling (ILFS) method. ILFS utilizes P4-based In-band Network Telemetry(INT) technology to collect fine-grained, real-time traffic data from SDIN's data plane. In the control plane, it integrates the Generative Adversarial Imitation Learning (GAIL) model with a Soft Actor Critic (SAC) to preserve the experiences of flow, thereby better scheduling large flows. Our experiments thoroughly compare ILFS's performance with several state-of-the-art traffic scheduling strategies. The results indicate that ILFS successfully controls the link bandwidth utilization between 10% and 80% and significantly improves the average network throughput and link utilization rate.

Synthesizing controlled microstructures of porous media using generative adversarial networks and reinforcement learning

For material modeling and discovery, synthetic microstructures play a critical role as digital twins. They provide stochastic samples upon which direct numerical simulations can be conducted to populate material databases. A large ensemble of simulation data on synthetic microstructures may provide supplemental data to inform and refine macroscopic material models, which might not be feasible from physical experiments alone. However, synthesizing realistic microstructures with realistic microstructural attributes is highly challenging. Thus, it is often oversimplified via rough approximations that may yield an inaccurate representation of the physical world. Here, we propose a novel deep learning method that can synthesize realistic three-dimensional microstructures with controlled structural properties using the combination of generative adversarial networks (GAN) and actor-critic (AC) reinforcement learning. The GAN-AC combination enables the generation of microstructures that not only resemble the appearances of real specimens but also yield user-defined physical quantities of interest (QoI). Our validation experiments confirm that the properties of synthetic microstructures generated by the GAN-AC framework are within a 5% error margin with respect to the target values. The scientific contribution of this paper resides in the novel design of the GAN-AC microstructure generator and the mathematical and algorithmic foundations therein. The proposed method will have a broad and substantive impact on the materials community by providing lenses for analyzing structure-property-performance linkages and for implementing the notion of ‘materials-by-design’.

Model-free dynamic control of robotic joints with integrated elastic ligaments

The combination of elasticity and rigidity found within mammalian limbs enables dexterous manipulation, agile, and versatile behavior, yet most modern robots are either primarily soft or rigid. Hybrid robots, composed of both soft and rigid parts, promote compliance to external forces while maintaining strength and stability provided by rigid robots. Most mammals have ligaments which connect bone to bone, enabling joints to passively redirect forces and softly constrain the range of motion. We present an approach to constructing a new class of hybrid joints through parametric design choices that adjust dynamic properties of the system. The inherent modularity and variability necessitate a model-free controller which can adjust to new contexts in relatively short time. Three joint examples are created along with three tasks to assess quality of the controllers, creating 9 total cases. We show the Soft Actor Critic (SAC) reinforcement learning algorithm outperforms a proportion–integral–derivative (PID) controller in 6/9 cases, yet this changes to 9/9 with a brief re-training period. This work presents a new class of hybrid robotic joints with modifiable dynamics and employs a model-free control training technique which can be fine-tuned for specific scenarios.