Action-value Function Research Articles

Research of Multi-agent Deep Reinforcement Learning based on Value Factorization

One of the numerous multi-agents’ deep reinforcements learning methods and a hotspot for research in the field is multi-agent deep reinforcement learning based on value factorization. In order to effectively address the issues of environmental instability and the exponential expansion of action space in multi-agent systems, it uses some constraints to break down the joint action value function of the multi-agent system into a specific combination of individual action value functions. Firstly, in this paper, the reason for the factorization of value function is explained. The fundamentals of multi-agent deep reinforcement learning are then introduced. The multi-agent deep reinforcement learning algorithms based on value factorization may then be separated into simple factorization and attention-mechanism based algorithms depending on whether other mechanisms are incorporated and which various mechanisms are introduced. Then several typical algorithms are introduced and their advantages and disadvantages are compared and analyzed. Finally, the content of reinforcement learning elaborated in this paper is summarized.

Dynamic Successor Features for transfer learning and guided exploration

The Successor Feature framework for Reinforcement Learning algorithms improves task transfer by decomposing the learned state–action value function. The decomposition involves two components, one that captures future-expected state features and the other that models the task-related reward structure. However, successful transfer between tasks depends heavily on how the reward function changes, possibly leading to failure of the original Successor Feature formulation. This paper proposes the Dynamic Successor Feature framework, DynSF, by extending the mathematical formulation of the original Successor Feature framework to center around a learned state-transition model. Under this formulation, the state-transition model dynamically induces the acting policy. The flexibility of DynSF also extends to the architecture, requiring only a state-transition model and a small vector of parameters. This architecture provides immense flexibility in the choice of the model used to learn the state-transition model. The DynSF framework is evaluated and compared to other baseline algorithms through several experiments in a continuous grid world environment, a robotic Reacher, and pixels in the Doom environment.

A Multi-Branch DQN-Based Transponder Resource Allocation Approach for Satellite Communications

In light of the increasing scarcity of frequency spectrum resources for satellite communication systems based on the transparent transponder, fast and efficient satellite resource allocation algorithms have become key to improving the overall resource occupancy. In this paper, we propose a reinforcement learning-based Multi-Branch Deep Q-Network (MBDQN), which introduces TL-Branch and RP-Branch to extract features of satellite resource pool state and task state simultaneously, and Value-Branch to calculate the action-value function. On the one hand, MBDQN improves the average resource occupancy performance (AOP) through the selection of multiple actions, including task selection and resource priority actions. On the other hand, the trained MBDQN is more suitable for online deployment and significantly reduces the runtime overhead due to the fact that MBDQN does not need iteration in the test phase. Experiments on both non-zero waste and zero waste datasets demonstrate that our proposed method achieves superior performance compared to the greedy or heuristic methods on the generated task datasets.

Reinforcement learning-based scheduling of multi-battery energy storage system

In this paper, a reinforcement learning-based multi-battery energy storage system (MBESS) scheduling policy is proposed to minimize the consumers' electricity cost. The MBESS scheduling problem is modeled as a Markov decision process (MDP) with unknown transition probability. However, the optimal value function is time-dependent and difficult to obtain because of the periodicity of the electricity price and residential load. Therefore, a series of time-independent action-value functions are proposed to describe every period of a day. To approximate every action-value function, a corresponding critic network is established, which is cascaded with other critic networks according to the time sequence. Then, the continuous management strategy is obtained from the related action network. Moreover, a two-stage learning protocol including offline and online learning stages is provided for detailed implementation in real-time battery management. Numerical experimental examples are given to demonstrate the effectiveness of the developed algorithm.

Dual-Timescale Resource Allocation for Collaborative Service Caching and Computation Offloading in IoT Systems

Edge computing has been envisioned as a key enabler to provide computation-intensive and delay-sensitive services in the future Internet of Things systems. By offloading the computational tasks to the edge server, both the service latency and energy consumption can be reduced. Since devices may request various types of computing services, caching appropriate services in the edge server to immediately provide computing resources can improve the quality of service. Nevertheless, it brings new challenges to jointly optimize the resource allocation, where the timeliness of caching and offloading operations are different. In this article, we first formulate the collaborative service caching and computation offloading as a dual-timescale resource allocation problem to minimize the costs of latency and energy consumption. Under this framework, a novel scheme based on hierarchical deep reinforcement learning is proposed to output collaborative caching and computing actions. Specifically, the proposed approach contains the service caching policy and the device computing policy with hierarchical action–value functions, which allows a flexible configuration of caching timescales. The simulation results demonstrate that the proposed policy outperforms the existing schemes on convergence performance and various parameters.

Sampled-data control through model-free reinforcement learning with effective experience replay

Reinforcement Learning (RL) based control algorithms can learn the control strategies for nonlinear and uncertain environment during interacting with it. Guided by the rewards generated by environment, a RL agent can learn the control strategy directly in a model-free way instead of investigating the dynamic model of the environment. In the paper, we propose the sampled-data RL control strategy to reduce the computational demand. In the sampled-data control strategy, the whole control system is of a hybrid structure, in which the plant is of continuous structure while the controller (RL agent) adopts a discrete structure. Given that the continuous states of the plant will be the input of the agent, the state–action value function is approximated by the fully connected feed-forward neural networks (FCFFNN). Instead of learning the controller at every step during the interaction with the environment, the learning and acting stages are decoupled to learn the control strategy more effectively through experience replay. In the acting stage, the most effective experience obtained during the interaction with the environment will be stored and during the learning stage, the stored experience will be replayed to customized times, which helps enhance the experience replay process.The effectiveness of proposed approach will be verified by simulation examples.

Deep reinforcement learning based energy management strategy for range extend fuel cell hybrid electric vehicle

To meet the power and long-range driving requirements of the vehicle, this paper presents a dual mode operation scheme for a range extend fuel cell hybrid vehicle for the first time, with an in-depth study of the pure electric mode and the range extend mode. The deep deterministic policy gradient algorithm is a well-known deep reinforcement learning algorithm that can solve complex nonlinear problems. To achieve the optimal power distribution among energy sources in the two modes, a dual deep deterministic policy gradient algorithm framework is proposed for the first time in this paper. In addition, a pervious action guidance mechanism is proposed to enable networks to approximate the action value function more efficiently in training. The training results show that the adopted previous action guidance mechanism helps to improve the learning convergence and exploration ability. The validation results show that the proposed strategy improves the operating economy by about 30% compared to the rule-based strategy, reduces the average fuel cell output fluctuation to less than 100 W, and reduces the fuel cell lifetime loss greatly. It is hoped that the proposed new structure, patterns, and energy management strategy will provide more ideas for scholars in future research.

Autonomous Maneuver Decision of UCAV Air Combat Based on Double Deep Q Network Algorithm and Stochastic Game Theory

Aiming at the problem that unmanned combat aerial vehicle (UCAV) is difficult to quickly and accurately perceive situation information and make maneuvering decision autonomously in modern air combat, which is easily affected by complex factors, a maneuvering decision algorithm of UCAV combined with deep reinforcement learning and game theory is proposed in this paper. Firstly, through the UCAV dynamics model and maneuver library, a reasonable air combat situation assessment model and advantage reward function are established, and the sample data of situation assessment indicators are constructed using the structure entropy weight method. Secondly, the convolutional neural network (CNN) is used to process the high-dimensional continuous situation features of UCAV in air combat, eliminate the correlation and redundancy between situation features, and train the neural network to approximate the action-value function. Then, the double deep Q network (DDQN) algorithm in reinforcement learning (RL) is introduced to train the agent by the interaction with the environment and combined with Minimax algorithm in stochastic game theory to solve the optimal value function in each specific state, and the optimal maneuver decision of UCAV is obtained. Air combat simulation results show that UCAV can choose maneuvers autonomously under different situations and occupy a dominant position quickly by this method, which greatly improves the combat effectiveness of UCAV.

Intelligent Pursuit–Evasion Game Based on Deep Reinforcement Learning for Hypersonic Vehicles

As defense technology develops, it is essential to study the pursuit–evasion (PE) game problem in hypersonic vehicles, especially in the situation where a head-on scenario is created. Under a head-on situation, the hypersonic vehicle’s speed advantage is offset. This paper, therefore, establishes the scenario and model for the two sides of attack and defense, using the twin delayed deep deterministic (TD3) gradient strategy, which has a faster convergence speed and reduces over-estimation. In view of the flight state–action value function, the decision framework for escape control based on the actor–critic method is constructed, and the solution method for a deep reinforcement learning model based on the TD3 gradient network is presented. Simulation results show that the proposed strategy enables the hypersonic vehicle to evade successfully, even under an adverse head-on scene. Moreover, the programmed maneuver strategy of the hypersonic vehicle is improved, transforming it into an intelligent maneuver strategy.

Adaptive attitude determination of bionic polarization integrated navigation system based on reinforcement learning strategy

<p style='text-indent:20px;'>The bionic polarization integrated navigation system includes three-axis gyroscopes, three-axis accelerometers, three-axis magnetometers, and polarization sensors, which provide pitch, roll, and yaw. When the magnetometers are interfered or the polarization sensors are obscured, the accuracy of attitude will be decreased due to abnormal measurement. To improve the accuracy of attitude of the integrated navigation system under these complex environments, an adaptive complementary filter based on DQN (Deep Q-learning Network) is proposed. The complementary filter is first designed to fuse the measurements from the gyroscopes, accelerometers, magnetometers, and polarization sensors. Then, a reward function of the bionic polarization integrated navigation system is defined as the function of the absolute value of the attitude angle error. The action-value function is introduced by a fully-connected network obtained by historical sensor data training. The strategy can be calculated by the deep Q-learning network and the action that optimal action-value function is obtained. Based on the optimized action, three types of integration are switched automatically to adapt to the different environments. Three cases of simulations are conducted to validate the effectiveness of the proposed algorithm. The results show that the adaptive attitude determination of bionic polarization integrated navigation system based on DQN can improve the accuracy of the attitude estimation.</p>

Fast Reinforcement Learning Based MPC based on NLP Sensitivities

This paper proposes a comprehensive approach to improve the computational efficiency of Reinforcement Learning (RL) based Model Predictive Controller (MPC). Although MPC will ensure controller safety and RL can generate optimal control policies, combining the two requires substantial time and computational effort, particularly for larger data sets. In a typical RL-based MPC and Q-learning workflow, two not-so-different MPC problems must be evaluated at each RL iteration, i.e. one for the action-value and one for the value function, which is time-consuming and prohibitively expensive in terms of computations. We employ nonlinear programming (NLP) sensitivities to approximate the action-value function using the optimal solution from the value function, reducing computational time. The proposed approach can achieve comparable performance to the conventional method but with significantly lower computational time. We demonstrate the proposed approach on two examples: Linear Quadratic Regulator (LQR) problem and Continuously Stirred Tank Reactor (CSTR).

Hierarchical and Stable Multiagent Reinforcement Learning for Cooperative Navigation Control.

We solve an important and challenging cooperative navigation control problem, Multiagent Navigation to Unassigned Multiple targets (MNUM) in unknown environments with minimal time and without collision. Conventional methods are based on multiagent path planning that requires building an environment map and expensive real-time path planning computations. In this article, we formulate MNUM as a stochastic game and devise a novel multiagent deep reinforcement learning (MADRL) algorithm to learn an end-to-end solution, which directly maps raw sensor data to control signals. Once learned, the policy can be deployed onto each agent, and thereby, the expensive online planning computations can be offloaded. However, to solve MNUM, traditional MADRL suffers from large policy solution space and nonstationary environment when agents make decisions independently and concurrently. Accordingly, we propose a hierarchical and stable MADRL algorithm. The hierarchical learning part introduces a two-layer policy model to reduce the solution space and uses an interlaced learning paradigm to learn two coupled policies. In the stable learning part, we propose to learn an extended action-value function that implicitly incorporates estimations of other agents' actions, based on which the environment's nonstationarity caused by other agents' changing policies can be alleviated. Extensive experiments demonstrate that our method can converge in a fast way and generate more efficient cooperative navigation policies than comparable methods.

Multi-Robot Learning Dynamic Obstacle Avoidance in Formation With Information-Directed Exploration

This paper presents an algorithm that generates distributed collision-free velocities for multi-robot while maintain formation as much as possible. The adaptive formation problem is cast as a sequential decision-making problem, which is solved using reinforcement learning that trains several distributed policies to avoid dynamic obstacles on the top of consensus velocities. We construct the policy with Bayesian Linear Regression based on a neural network (called BNL) to compute the state-action value uncertainty efficiently for sequential decision making. The information-directed sampling is applied in our BNL policy to achieve efficient exploration. By further combining the distributional reinforcement learning, we can estimate the intrinsic uncertainty of the state-action value globally and more accurately. For continuous control tasks, efficient exploration can be achieved by optimizing a policy with the sampled action value function from a BNL model. Through our experiments in some contextual Bandit and sequential decision-making tasks, we show that exploration with the BNL model has improved efficiency in both computation and training samples. By augmenting the consensus velocities with our BNL policy, experiments on Multi-Robot navigation demonstrate that adaptive formation is achieved.

Autonomous trajectory planning method for hypersonic vehicles in glide phase based on DDPG algorithm

An autonomous optimal trajectory planning method based on the deep deterministic policy gradient (DDPG) algorithm of reinforcement learning (RL) for hypersonic vehicles (HV) is proposed in this paper. First, the trajectory planning problem is converted into a Markov Decision Process (MDP), and the amplitude of the bank angle is designated as the control input. The reward function of the MDP is set to minimize the trajectory terminal position errors with satisfying hard constraints. The deep neural network (DNN) is used to approximate the policy function and action-value function in the DDPG framework. The Actor network then computes the control input directly according to flight states. Using a limited exploration strategy, the optimal policy network would be considered fully trained when the reward value reached maximum convergence. Simulation results show that the policy network trained using a DDPG algorithm accomplishes 3-dimensional (3D) trajectory planning during the HV glide phase with high terminal precision and stable convergence. Additionally, the single step calculation time of the policy network occurs in near real time, which suggests great potential as an autonomous online trajectory planner. Monte Carlo experiments prove the strong robustness of the implementation of an autonomous trajectory planner under aerodynamic disturbances.

A Hybrid Data-Driven Method for Fast Solution of Security-Constrained Optimal Power Flow

This paper proposes a hybrid data-driven method for fast solutions of preventive security-constrained optimal power flow (SCOPF) of power systems. The proposed method formulates the SCOPF problem as constraints-satisfying training of a deep reinforcement learning (DRL) agent, where the action-value function of DRL is augmented by contingency security constraints. In the training process, the proposed method hybridizes the primal-dual deep deterministic policy gradient (PD-DDPG) and the classic SCOPF model. Instead of building reward critic networks and cost critic networks via interacting with the environment (i.e., power flow equations), the actor gradients are approximated by solving KKT conditions of the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Lagrangian</i> . Finally, with the formulated sparse Jacobians of constraints and sparse Hessians of Lagrangians, the interior point method is incorporated in PD-DDPG to derive the parameters updating rule of the DRL agent. Numerical tests are carried out on a modified IEEE 57-bus system and a modified IEEE 300-bus system for critical contingencies. The results show that the well-trained DRL agent can rapidly (real-time) obtain high-quality SCOPF solutions that satisfy the security constraints.

Joint Channel and Power Assignment for UAV Swarm Communication Based on Multi-Agent DRL

In this manuscript, we propose a joint channel and power assignment algorithm for an unmanned aerial vehicle (UAV) swarm communication system based on multi-agent deep reinforcement learning (DRL). Regarded as an agent, each UAV to UAV (U2U) link can choose the optimal channel and power according to the current situation after training is successfully completed. Further, a mixing network is introduced based on DRL, where Q values of every single agent are non-linearly mapped, and we call it the QMIX algorithm. As it accesses state information, QMIX can learn to enrich the joint action value function. The proposed method can be used for both unicast and multicast scenarios. Experiments show that each U2U link can be trained to meet the constraints of UAV communication and minimize the interference to the system. For unicast communication, the communication rate is increased up to 15.6% and 8.9% using the proposed DRL method compared with the well-known random and adaptive methods, respectively. For multicast communication, the communication rate is increased up to 6.7% using the proposed QMIX method compared with the DRL method and 13.6% using DRL method compared with adaptive method. Besides, the successful transmission probability can maintain a high level.

Simulation Design of a Live Working Manipulator for Patrol Inspection in Power Grid

The distribution line network is the electric power infrastructure directly facing the users, with the characteristics of large coverage and complex network, and its operation safety is directly related to the stability and reliability of the power supply system, which is the key link to ensure the safety of power supply and the reliability of residential electricity consumption. In order to realize the autonomous obstacle avoidance and autonomous navigation of the live working manipulator for inspection and maintenance of the power grid equipment, a mobile manipulator intelligent control method combining SumTree-weighted sampling and deep deterministic policy gradient (DDPG) is proposed. Firstly, the traditional DDPG algorithm is improved to optimize the action value function of Q-learning to get a better control strategy, and the weighted sampling technique is used to add priority to each sample in the replay buffer, which improves the learning speed and accelerates the convergence speed. The corresponding environmental state space is designed, and simulation experiments are conducted to verify the proposed manipulator control method. Simulation results demonstrate that the proposed method performs better than traditional DDPG and DQN methods in obstacle avoidance and navigation tasks, with faster convergence, better path planning ability, and lower offset cost, which can provide theoretical and technical references for realizing fully autonomous power grid inspection operations.

Q-learning of the storage function in Economic Nonlinear Model Predictive Control

The closed-loop stability of an optimal policy provided by an Economic Nonlinear Model Predictive Control (ENMPC) scheme requires the existence of a storage function satisfying dissipativity conditions. Unfortunately, finding such a storage function is difficult in general. In contrast, tracking NMPC scheme uses a stage cost that is lower-bounded by a class-K∞ function and the closed-loop stability is fairly straightforward to establish. Under the dissipativity conditions, ENMPC has an equivalent tracking MPC that delivers the same optimal policy. In this paper, we use this idea and parameterize the stage cost and terminal cost of a tracking MPC with an additional parameterized storage function. We show that, if the parameterization of the tracking MPC scheme is rich enough to capture the exact optimal action-value function of the ENMPC scheme, then the parameterized storage function for the optimal parameters satisfies the dissipativity conditions for both discounted and undiscounted ENMPC schemes. In fact, we show that these conditions are met for dissipative problems. We propose to use Q-learning as a practical way of adjusting the parameters of the tracking MPC. Different numerical examples are provided to illustrate the efficiency of the proposed method, including LQR, non-dissipative, non-polynomial and a nonlinear chemical case studies. For instance, in the provided non-polynomial case study, the learning method can improve the storage function estimation by about 60% and 99.5% after 10 and 50 learning steps, respectively, compared with the Sum-of-Square method.

Multirobot Coverage Path Planning Based on Deep Q-Network in Unknown Environment

Aiming at the problems of security, high repetition rate, and many restrictions of multirobot coverage path planning (MCPP) in an unknown environment, Deep Q-Network (DQN) is selected as a part of the method in this paper after considering its powerful approximation ability to the optimal action value function. Then, a deduction method and some environments handling methods are proposed to improve the performance of the decision-making stage. The deduction method assumes the movement direction of each robot and counts the reward value obtained by the robots in this way and then determines the actual movement directions combined with DQN. For these reasons, the whole algorithm is divided into two parts: offline training and online decision-making. Online decision-making relies on the sliding-view method and probability statistics to deal with the nonstandard size and unknown environments and the deduction method to improve the efficiency of coverage. Simulation results show that the performance of the proposed online method is close to that of the offline algorithm which needs long time optimization, and the proposed method is more stable as well. Some performance defects of current MCPP methods in an unknown environment are ameliorated in this study.

A policy iteration method for improving robot assembly trajectory efficiency

Bolt assembly by robots is a vital and difficult task for replacing astronauts in extra-vehicular activities (EVA), but the trajectory efficiency still needs to be improved during the wrench insertion into hex hole of bolt. In this paper, a policy iteration method based on reinforcement learning (RL) is proposed, by which the problem of trajectory efficiency improvement is constructed as an issue of RL-based objective optimization. Firstly, the projection relation between raw data and state-action space is established, and then a policy iteration initialization method is designed based on the projection to provide the initialization policy for iteration. Policy iteration based on the protective policy is applied to continuously evaluating and optimizing the action-value function of all state-action pairs till the convergence is obtained. To verify the feasibility and effectiveness of the proposed method, a noncontact demonstration experiment with human supervision is performed. Experimental results show that the initialization policy and the generated policy can be obtained by the policy iteration method in a limited number of demonstrations. A comparison between the experiments with two different assembly tolerances shows that the convergent generated policy possesses higher trajectory efficiency than the conservative one. In addition, this method can ensure safety during the training process and improve utilization efficiency of demonstration data.