Actor-critic Framework Research Articles

Online Optimal Adaptive Control of Partially Uncertain Nonlinear Discrete-Time Systems Using Multilayer Neural Networks.

This article intends to address an online optimal adaptive regulation of nonlinear discrete-time systems in affine form and with partially uncertain dynamics using a multilayer neural network (MNN). The actor-critic framework estimates both the optimal control input and value function. Instantaneous control input error and temporal difference are used to tune the weights of the critic and actor networks, respectively. The selection of the basis functions and their derivatives are not required in the proposed approach. The state vector, critic, and actor NN weights are proven to be bounded using the Lyapunov method. Our approach can be extended to neural networks with an arbitrary number of hidden layers. We have demonstrated our approach via a simulation example.

Intelligent Reflecting Surface-Aided Device-to-Device Communication: A Deep Reinforcement Learning Approach

Recently, the growing demand of various emerging applications in the realms of sixth-generation (6G) wireless networks has made the term internet of Things (IoT) very popular. Device-to-device (D2D) communication has emerged as one of the significant enablers for the 6G-based IoT network. Recently, the intelligent reflecting surface (IRS) has been considered as a hardware-efficient innovative scheme for future wireless networks due to its ability to mitigate propagation-induced impairments and to realize a smart radio environment. Such an IRS-assisted D2D underlay cellular network is investigated in this paper. Our aim is to maximize the network’s spectrum efficiency (SE) by jointly optimizing the transmit power of both the cellular users (CUs) and the D2D pairs, the resource reuse indicators, and the IRS reflection coefficients. Instead of using traditional optimization solution schemes to solve this mixed integer nonlinear optimization problem, a reinforcement learning (RL) approach is used in this paper. The IRS-assisted D2D communication network is structured by the Markov Decision Process (MDP) in the RL framework. First, a Q-learning-based solution is studied. Then, to make a scalable solution with large dimension state and action spaces, a deep Q-learning-based solution scheme using experience replay is proposed. Lastly, an actor-critic framework based on the deep deterministic policy gradient (DDPG) scheme is proposed to learn the optimal policy of the constructed optimization problem considering continuous-valued state and action spaces. Simulation outcomes reveal that the proposed RL-based solution schemes can provide significant SE enhancements compared to the existing optimization schemes.

A Rear Anti-Collision Decision-Making Methodology Based on Deep Reinforcement Learning for Autonomous Commercial Vehicles

Driving decision-making determines the safety and rationality of autonomous commercial vehicles. Aiming at the issue of safe driving decision-making, herein, a rear anti-collision decision-making methodology based on deep reinforcement learning (RAD-DRL) was creatively proposed. Firstly, aiming at the dynamic coupling of rear anti-collision factors for safe driving, a driving decision network based on an actor-critic framework was proposed for sensor data processing. Then, inspired by the idea of multi-objective optimization, a refined reward function is developed. It comprehensively considers the impact of backward target types, safety clearance, and vehicle roll stability on the rear collision. Finally, the RAD-DRL was trained (with different values of random seeds), tested, and verified in a simulation environment where road network and traffic situations were built-in SUMO (Simulation of Urban Mobility). After 30,000 training episodes, effective and reliable rear anti-collision driving decisions were achieved by our proposed RAD-DRL. The simulated results indicated that the RAD-DRL has remarkable superiority in generalization and effectiveness in expressway scenarios with different driving cases. Especially, it maintained good decision performance in the corner case in which the backward vehicle suddenly accelerates.

Deconfounding Actor-Critic Network with Policy Adaptation for Dynamic Treatment Regimes.

Despite intense efforts in basic and clinical research, an individualized ventilation strategy for critically ill patients remains a major challenge. Recently, dynamic treatment regime (DTR) with reinforcement learning (RL) on electronic health records (EHR) has attracted interest from both the healthcare industry and machine learning research community. However, most learned DTR policies might be biased due to the existence of confounders. Although some treatment actions non-survivors received may be helpful, if confounders cause the mortality, the training of RL models guided by long-term outcomes (e.g., 90-day mortality) would punish those treatment actions causing the learned DTR policies to be suboptimal. In this study, we develop a new deconfounding actor-critic network (DAC) to learn optimal DTR policies for patients. To alleviate confounding issues, we incorporate a patient resampling module and a confounding balance module into our actor-critic framework. To avoid punishing the effective treatment actions non-survivors received, we design a short-term reward to capture patients' immediate health state changes. Combining short-term with long-term rewards could further improve the model performance. Moreover, we introduce a policy adaptation method to successfully transfer the learned model to new-source small-scale datasets. The experimental results on one semi-synthetic and two different real-world datasets show the proposed model outperforms the state-of-the-art models. The proposed model provides individualized treatment decisions for mechanical ventilation that could improve patient outcomes.

Constrained Soft Actor-Critic for Energy-Aware Trajectory Design in UAV-Aided IoT Networks

This letter investigates an unmanned aerial vehicle (UAV)-aided data collection system, where a UAV is deployed to gather information from terrestrial Internet of Things (IoT) devices. We aim to minimize the mission completion time by optimizing the UAV trajectory, while ensuring all the target data can be successfully collected with a given energy budget. Firstly, the problem is formulated as a constrained Markov Decision Process (CMDP). Then, a constrained soft actor-critic (CSAC) algorithm is proposed by incorporating Lagrangian primal-dual optimization (PDO) into the soft actor-critic (SAC) framework. Simulation results demonstrate that the proposed algorithm outperforms state-of-the-art benchmark algorithms in terms of the mission completion time. Particularly, it is able to learn an adaptive policy that outputs optimal trajectories for different device locations.

A Proposed Priority Pushing and Grasping Strategy Based on an Improved Actor-Critic Algorithm

The most basic and primary skills of a robot are pushing and grasping. In cluttered scenes, push to make room for arms and fingers to grasp objects. We propose a modified Actor-Critic (A-C) framework for deep reinforcement learning, Cross-entropy Softmax A-C (CSAC), and use the Prioritized Experience Replay (PER) based on the theoretical foundation and main methods of deep reinforcement learning, combining the advantages of algorithms based on value functions and policy gradients. The grasping model is trained using self-supervised learning to achieve end-to-end mapping from image to propulsion and grasping action. A vision module and an action module have been created out of the entire algorithm framework. The prioritized experience replay is improved to further improve the CSAC-PER algorithm for model sample diversity and robot exploration performance during robot grasping training. The experience replay buffer is dynamically sampled using the prior beta distribution and the dynamic sampling algorithm based on the beta distribution (CSAC-β) is proposed based on the CSAC algorithm. Despite its low initial efficiency, the experimental simulation results show that the CSAC-β algorithm eventually achieves good results and has a higher grasping success rate (90%).

Lightweight actor-critic generative adversarial networks for real-time smart generation control of microgrids

Large-scale introduction of new energy could effectively alleviate energy shortage and environmental pollution. However, the uncertainty of wind and solar energy brings serious random disturbance problems to microgrids. This paper proposes lightweight actor-critic generative adversarial networks based on ensemble empirical mode decomposition and evolutionary strategy for increasing the robustness and adaptability of microgrids. Firstly, to improve the training speed and stability of generative adversarial networks, the complex power data is properly decomposed into more regular and simpler sub-data by the ensemble empirical mode decomposition; the generative adversarial networks are optimized by the evolutionary strategy with a set of different loss functions. Secondly, fully connecting the generative adversarial networks with the actor-critic framework, the lightweight actor-critic generative adversarial networks can realize dynamic learning in the random environment and store the sample for online training by the empirical replay mechanism. Thirdly, the multi-path lightweight method is proposed to reduce the consumption of time and storage resources of lightweight actor-critic generative adversarial networks. Eventually, the lightweight actor-critic generative adversarial networks are compared with comparison algorithms in two-area and real-life four-area systems. Case study results reveal that lightweight actor-critic generative adversarial networks have better dynamic performance, online learning capabilities, and higher control performance with lower economic costs.

Transient Controller Design Based on Reinforcement Learning for a Turbofan Engine with Actuator Dynamics

To solve the problem of transient control design with uncertainties and degradation in the life cycle, a design method for a turbofan engine’s transient controller based on reinforcement learning is proposed. The method adopts an actor–critic framework and deep deterministic policy gradient (DDPG) algorithm with the ability to train an agent with continuous action policy for the continuous and violent turbofan engine state change. Combined with a symmetrical acceleration and deceleration transient control plan, a reward function with the aim of servo tracking is proposed. Simulations under different conditions were carried out with a controller designed via the proposed method. The simulation results show that during the acceleration process of the engine from idle to an intermediate state, the controlled variables have no overshoot, and the settling time does not exceed 3.8 s. During the deceleration process of the engine from an intermediate state to idle, the corrected speed of high-pressure rotor has no overshoot, the corrected-speed overshoot of the low-pressure rotor does not exceed 1.5%, and the settling time does not exceed 3.3 s. A system with the designed transient controller can maintain the performance when uncertainties and degradation are considered.

Computation Offloading in Multi-UAV-Enhanced Mobile Edge Networks: A Deep Reinforcement Learning Approach

In this paper, we investigate an unmanned aerial vehicle- (UAV-) enhanced mobile edge computing network (MUEMN), where multiple UAVs are deployed as aerial edge servers to provide computing services for ground moving equipment (GME). Each GME is trained to simulate movement by a Gauss-Markov random model in this MUEMN. Under the condition of limited energy cost, UAV dynamically plans its flight position according to the movement trend of GME. Our objective is to minimize the total energy consumption of GME by jointly optimizing the offloading decisions of GME and the flight positions of UAVs. More explicitly, we model the optimization problem as a Markov decision process and achieve real-time offloading decisions via deep reinforcement learning algorithm according to the dynamic system state, where the asynchronous advantage actor-critic (A3C) framework with asynchronous characteristics is leveraged to accelerate the learning process. Finally, numerical results confirm that our proposed A3C-based offloading strategy can effectively reduce the total of energy consumption of GME and ensure the continuous operation of the GME.

PID Control of Permanent Magnet Synchronous Motor Based on Improved Actor-Critic Framework

Aiming at the shortcomings of the traditional PID control method that the parameters cannot be adjusted flexible, an improved Actor-Critic reinforcement learning algorithm combined with incremental PID control is proposed to improve the control performance of permanent magnet synchronous motor (PMSM). The strategy function of Actor and value function of Critic are approximated by two back propagation (BP) neural networks respectively. The simulation results show that the proposed algorithm has better control performance and effect than the traditional PID control method.

Modelling personalised car-following behaviour: a memory-based deep reinforcement learning approach

To adapt to human-driving habits, this study develops a personalised car-following model via a memory-based deep reinforcement learning approach. Specifically, Twin Delayed Deep Deterministic Policy Gradients (TD3) is integrated with a long short-term memory (LSTM) (abbreviated as LSTM-TD3). Using the NGSIM dataset, unsupervised learning-based clustering and data feature analyses are performed. The driving characteristics related to safety, efficiency and comfort are extracted for different driving styles, i.e. aggressive, common and conservative. Then, reward functions are constructed for different driving styles by incorporating their driving characteristics. By resorting to the TD3 policy within a recurrent actor–critic framework, LSTM-TD3 optimises the car-following behaviour via trial-and-error interactions according to the reward functions. Results show that compared with LSTM-DDPG and DDPG, LSTM-TD3 reproduces personalised car-following behaviour with desirable convergence speed and reward. It reveals that LSTM-TD3 can reflect the essential difference in safety, efficiency and comfort requirements among different driving styles.

A soft actor-critic-based energy management strategy for electric vehicles with hybrid energy storage systems

With the rapid development of machine learning, deep reinforcement learning (DRL) algorithms have been widely applied to energy management strategies (EMSs) of hybrid vehicles recently. However, current DRL algorithms show the drawbacks of slower convergence rate, brittle training stability, and dissatisfactory optimization effects. In this research, a new DRL algorithm, i.e. the soft actor-critic (SAC) is applied to the EMS of an electric vehicle (EV) with a hybrid energy storage system (HESS). Particularly, the knowledge extracted from the dynamic programming (DP), which is regarded as the benchmark for control methods, is adopted to improve the control performance of the SAC-based EMS and the parallel computing is applied to accelerate the training process. Results of this research indicate that the SAC-based EMS decreases the HESS energy loss by 8.75% and 6.09% compared to the deep Q-network (DQN) and deep deterministic policy gradient (DDPG)-based EMSs respectively while it narrows the gap with the DP-based EMS to 5.19%. Additionally, as the same actor-critic framework, the convergence rate of the SAC-based EMS is faster than that of the DDPG-based EMS by 205.66%. Furthermore, the adaptability validation results present that the SAC-based EMS outperforms the DQN and DDPG-based EMSs in energy saving up to 32.24%.

Parameter identification of energy consumption dynamic model for double-motor-driven belt conveyers based on Actor–Critic framework

Parameter identification of energy consumption dynamic model for double-motor-driven belt conveyers based on Actor–Critic framework

TRACE: Travel Reinforcement Recommendation Based on Location-Aware Context Extraction

As the popularity of online travel platforms increases, users tend to make ad-hoc decisions on places to visit rather than preparing the detailed tour plans in advance. Under the situation of timeliness and uncertainty of users’ demand, how to integrate real-time context into dynamic and personalized recommendations have become a key issue in travel recommender system. In this article, by integrating the users’ historical preferences and real-time context, a location-aware recommender system called TRACE ( T ravel R einforcement Recommendations Based on Location- A ware C ontext E xtraction) is proposed. It captures users’ features based on location-aware context learning model, and makes dynamic recommendations based on reinforcement learning. Specifically, this research: (1) designs a travel reinforcing recommender system based on an Actor-Critic framework, which can dynamically track the user preference shifts and optimize the recommender system performance; (2) proposes a location-aware context learning model, which aims at extracting user context from real-time location and then calculating the impacts of nearby attractions on users’ preferences; and (3) conducts both offline and online experiments. Our proposed model achieves the best performance in both of the two experiments, which demonstrates that tracking the users’ preference shifts based on real-time location is valuable for improving the recommendation results.

Drift-Proof Tracking With Deep Reinforcement Learning

Object tracking is an essential and challenging sub-domain in the field of computer vision owing to its wide range of applications and complexities of real-life situations. It has been studied extensively over the last decade, leading to the proposal of several tracking frameworks and approaches. Recently, the introduction of reinforcement learning and the Actor-Critic framework has effectively improved the tracking speed of deep learning trackers. However, most existing deep reinforcement learning trackers experience a slight performance degradation mainly owing to the drift issues. Drifts pose a threat to the tracking performance, which may lead to losing the tracked target. Herein, we propose a drift-proof tracker with deep reinforcement learning that aims to improve the tracking performance by counteracting drifts while maintaining its real-time advantage. We utilize a reward function with the Distance-IoU (DIoU) metric to guide the reinforcement learning to alleviate the drifts caused by the trained model. Furthermore, double negative samples (hard negative and drift samples) are constructed in tracking for network initialization, which is followed by calculating the loss by a small error-friendly loss function. Therefore, our tracker can better discriminate between the positive and negative samples and correct the predicted bounding boxes when the drift occurs. Meanwhile, a generative adversarial network is adopted for positive sample augmentation. Extensive experimental results on multiple popular benchmarks show that our algorithm effectively reduces the occurrences of drift and boosts the tracking performance, compared to those of other state-of-the-art trackers.

Sim2real Learning of Obstacle Avoidance for Robotic Manipulators in Uncertain Environments

Obstacle avoidance for robotic manipulators can be challenging when they operate in unstructured environments. This problem is probed with the sim-to-real (sim2real) deep reinforcement learning, such that a moving policy of the robotic arm is learnt in a simulator and then adapted to the real world. However, the problem of sim2real adaptation is notoriously difficult. To this end, this work proposes (1) a unified representation of obstacles and targets to capture the underlying dynamics of the environment while allowing generalization to unseen goals and (2) a flexible end-to-end model combining the unified representation with the deep reinforcement learning control module that can be trained by interacting with the environment. Such a representation is agnostic to the shape and appearance of the underlying objects, which simplifies and unifies the scene representation in both simulated and real worlds. We implement this idea with a vision-based actor-critic framework by devising a bounding box predictor module. The predictor estimates the 3D bounding boxes of obstacles and targets from the RGB-D input. The features extracted by the predictor are fed into the policy network, and all the modules are jointly trained. This makes the policy learn object-aware scene representation, which leads to a data-efficient learning of the obstacle avoidance policy. Our experiments in simulated environment and the real-world show that the end-to-end model of the unified representation achieves better sim2real adaption and scene generalization than state-of-the-art techniques.

Learning practically feasible policies for online 3D bin packing

We tackle the online 3D bin packing problem (3D-BPP), a challenging yet practically useful variant of the classical bin packing problem. In this problem, the items are delivered to the agent without informing the full sequence information. The agent must directly pack these items into the target bin stably without changing their arrival order, and no further adjustment is permitted. Online 3D-BPP can be naturally formulated as a Markov decision process (MDP). We adopt deep reinforcement learning, in particular, the on-policy actor-critic framework, to solve this MDP with constrained action space. To learn a practically feasible packing policy, we propose three critical designs. First, we propose an online analysis of packing stability based on a novel stacking tree. It attains a high analysis accuracy while reducing the computational complexity from O(N2) to O(N log N), making it especially suited for reinforcement learning training. Second, we propose a decoupled packing policy learning for different dimensions of placement which enables high-resolution spatial discretization and hence high packing precision. Third, we introduce a reward function that dictates the robot to place items in a far-to-near order and therefore simplifies the collision avoidance in movement planning of the robotic arm. Furthermore, we provide a comprehensive discussion on several key implemental issues. The extensive evaluation demonstrates that our learned policy outperforms the state-of-the-art methods significantly and is practically usable for real-world applications.

Acceleration control strategy for aero-engines based on model-free deep reinforcement learning method

Deep reinforcement learning has emerged as a powerful control method, especially for the complex nonlinear system such as aeroengine control system, due to its strong representation ability and capability of learning from data measurements. This paper presents a novel control strategy based on deep reinforcement learning to speed up the acceleration process of aero-engine. The actor-critic framework is adopted, where the actor neural network aims to find the optimal control policy and the critic network aims to evaluate the current control policy. The deep deterministic policy gradient algorithm is used to update the parameters of the neural networks. In addition, a complementary integrator is introduced to eliminate the steady-state error caused by the approximation error of the deep neural networks, and a momentum term is introduced to set limits for the input of the control system, thereby suppressing the overrun during the early learning and exploration processes. The numerical simulations show that the controller with this new control strategy can cope with different flight conditions, and significantly speed up the acceleration process of aero-engine.

Multi actor hierarchical attention critic with RNN-based feature extraction

Deep reinforcement learning has made significant progress in multi-agent tasks in recent years. However, most previous studies focus on solving fully cooperative tasks, which do not perform well in mixed tasks. In mixed tasks, the agent needs to comprehensively consider the information provided by its friends and enemies to learn its strategy, and its strategy is sensitive to the received information. Additionally, the input space of the critic network increases rapidly with the number of agents in the actor-critic framework. It’s of great necessity to efficiently learn information representation to obtain important features. To this end, we present an approach that conducts information representation with attention mechanism. Our approach adopts the framework of centralized training and decentralized execution. We apply the multi-head hierarchical attention mechanism to centrally computed critics, so critics can process the received information more accurately and assist actors in choosing better actions. The hierarchical attention critic adopts a bi-level attention structure which is composed of the agent-level and the group-level. They are designed to assign different weights to friends’ and enemies’ information and then summarize them at each timestep. It achieves high efficiency and scalability in mixed tasks. Furthermore, we use the feature extraction based on the recurrent neural network to encode the state-action sequence information of each agent. Experimental results show that our approach is not only applicable to cooperative environments but also better in mixed environments, especially in the predator-prey task, the reward obtained by our method is twice that of the baselines.

Deep Reinforcement Learning for Continuous Electric Vehicles Charging Control With Dynamic User Behaviors

This paper aims to crack the individual EV charging scheduling problem considering the dynamic user behaviors and the electricity price. The uncertainty of the EV charging demand is described by several factors, including the driver’s experience, the charging preference and the charging locations for realistic scenarios. An aggregate anxiety concept is introduced to characterize both the driver’s anxiety on the EV’s range and uncertain events. A mathematical model is also provided to describe the anxiety quantitatively. The problem is formulated as a Markov Decision Process (MDP) with an unknown state transition function. The objective is to find the optimal sequential charging decisions that can balance the charging cost and driver’s anxiety. A model-free deep reinforcement learning (DRL) based approach is developed to learn the optimal charging control strategy by interacting with the dynamic environment. The continuous soft actor-critic (SAC) framework is applied to design the learning method, which contains a supervised learning (SL) stage and a reinforcement learning (RL) stage. Finally, simulation studies verify the effectiveness of the proposed approach under dynamic user behaviors at different charging locations.