Continuous Action Spaces Research Articles

Deep reinforcement learning for real-time economic energy management of microgrid system considering uncertainties

The electric power grid is changing from a traditional power system to a modern, smart, and integrated power system. Microgrids (MGs) play a vital role in combining distributed renewable energy resources (RESs) with traditional electric power systems. Intermittency, randomness, and volatility constitute the disadvantages of distributed RESs. MGs with high penetrations of renewable energy and random load demand cannot ignore these uncertainties, making it difficult to operate them effectively and economically. To realize the optimal scheduling of MGs, a real-time economic energy management strategy based on deep reinforcement learning (DRL) is proposed in this paper. Different from traditional model-based approaches, this strategy is learning based, and it has no requirements for an explicit model of uncertainty. Taking into account the uncertainties in RESs, load demand, and electricity prices, we formulate a Markov decision process for the real-time economic energy management problem of MGs. The objective is to minimize the daily operating cost of the system by scheduling controllable distributed generators and energy storage systems. In this paper, a deep deterministic policy gradient (DDPG) is introduced as a method for resolving the Markov decision process. The DDPG is a novel policy-based DRL approach with continuous state and action spaces. The DDPG is trained to learn the characteristics of uncertainties of the load, RES output, and electricity price using historical data from real power systems. The effectiveness of the proposed approach is validated through the designed simulation experiments. In the second experiment of our designed simulation, the proposed DRL method is compared to DQN, SAC, PPO, and MPC methods, and it is able to reduce the operating costs by 29.59%, 17.39%, 6.36%, and 9.55% on the June test set and 30.96%, 18.34%, 5.73%, and 10.16% on the November test set, respectively. The numerical results validate the practical value of the proposed DRL algorithm in addressing economic operation issues in MGs, as it demonstrates the algorithm’s ability to effectively leverage the energy storage system to reduce the operating costs across a range of scenarios.

Deep Reinforcement Learning-Based Control of Stewart Platform With Parametric Simulation in ROS and Gazebo

AbstractThe Stewart platform is an entirely parallel robot with mechanical differences from typical serial robotic manipulators, which has a wide application area ranging from flight and driving simulators to structural test platforms. This work concentrates on learning to control a complex model of the Stewart platform using state-of-the-art deep reinforcement learning (DRL) algorithms. In this regard, to enhance the reliability of the learning performance and to have a test bed capable of mimicking the behavior of the system completely, a precisely designed simulation environment is presented. Therefore, we first design a parametric representation for the kinematics of the Stewart platform in Gazebo and robot operating system (ROS) and integrate it with a Python class to conveniently generate the structures in simulation description format (SDF). Then, to control the system, we benefit from three DRL algorithms: the asynchronous advantage actor–critic (A3C), the deep deterministic policy gradient (DDPG), and the proximal policy optimization (PPO) to learn the control gains of a proportional integral derivative (PID) controller for a given reaching task. We chose to apply these algorithms due to the Stewart platform’s continuous action and state spaces, making them well-suited for our problem, where exact controller tuning is a crucial task. The simulation results show that the DRL algorithms can successfully learn the controller gains, resulting in satisfactory control performance.

A Task-Agnostic Regularizer for Diverse Subpolicy Discovery in Hierarchical Reinforcement Learning

The automatic subpolicy discovery approach in hierarchical reinforcement learning (HRL) has recently achieved promising performance on sparse reward tasks. This accelerates transfer learning and unsupervised intelligent creatures while eliminating the domain-specific knowledge constraint. Most previously developed approaches are demonstrated to suffer from collapsing into the situation where one subpolicy dominates the whole task, since they cannot ensure the diversity of different subpolicies. In contrast, this article proposes a task-agnostic regularizer (TAR) for learning diverse subpolicies in HRL. Specifically, we first formulate the discovery of diverse subpolicies as a trajectory inference problem and then propose a corresponding information-theoretic objective to encourage diversity. Subsequently, considering computability, we instantiate the objective as two simplifications for discrete and continuous action spaces. We extensively evaluate the proposed diversity-driven regularizer on three HRL task domains: 1) meta reinforcement learning; 2) hierarchical policy learning in the option framework; and 3) unsupervised subpolicy discovery. The extensive results obtained show that our TAR approach can improve upon the state-of-the-art performance on all three HRL domains without modifying any existing hyperparameters, indicating the wide applicability and robustness of our approach.

Opinion Dynamics in Gossiper-Media Networks Based on Multiagent Reinforcement Learning

In social networks, to increase the number of followers, media express their opinions through posts, articles, and published content to cater to public preferences. Meanwhile, the evolution of public opinions is affected by both the media and peers. In this work, we investigate how the interactions between media affect the dynamics of public opinions in social networks. We propose a reinforcement learning framework to model the interactions between the public (gossipers) and media agents. We model each gossiper as an individually rational agent, which updates its opinion using the bounded confidence model (BCM). Media agents are interested in maximizing the number of following gossipers competitively, and an algorithm, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e</i> ., WoLS-CALA, is proposed to achieve that goal. We analyze and experimentally verify that WoLS-CALA can learn Nash equilibria (NE) for games with continuous action space. In addition, the opinion dynamics of gossipers are theoretically analyzed, which shows that the existence of media will strengthen the consistency of gossipers' opinions. We then evaluate the framework in two synthetic networks, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e</i> ., a fully connected network and a small-world network, and one real data network from Facebook. Extensive empirical simulation reveals that our framework facilitates the consensus of opinions and confirms the theoretical analysis.

Modeling Human Driving Behavior Through Generative Adversarial Imitation Learning

An open problem in autonomous vehicle safety validation is building reliable models of human driving behavior in simulation. This work presents an approach to learn neural driving policies from real world driving demonstration data. We model human driving as a sequential decision making problem that is characterized by non-linearity and stochasticity, and unknown underlying cost functions. Imitation learning is an approach for generating intelligent behavior when the cost function is unknown or difficult to specify. Building upon work in inverse reinforcement learning (IRL), Generative Adversarial Imitation Learning (GAIL) aims to provide effective imitation even for problems with large or continuous state and action spaces, such as modeling human driving. This article describes the use of GAIL for learning-based driver modeling. Because driver modeling is inherently a multi-agent problem, where the interaction between agents needs to be modeled, this paper describes a parameter-sharing extension of GAIL called PS-GAIL to tackle multi-agent driver modeling. In addition, GAIL is domain agnostic, making it difficult to encode specific knowledge relevant to driving in the learning process. This paper describes Reward Augmented Imitation Learning (RAIL), which modifies the reward signal to provide domain-specific knowledge to the agent. Finally, human demonstrations are dependent upon latent factors that may not be captured by GAIL. This paper describes Burn-InfoGAIL, which allows for disentanglement of latent variability in demonstrations. Imitation learning experiments are performed using NGSIM, a real-world highway driving dataset. Experiments show that these modifications to GAIL can successfully model highway driving behavior, accurately replicating human demonstrations and generating realistic, emergent behavior in the traffic flow arising from the interaction between driving agents.

A deep Koopman operator‐based modelling approach for long‐term prediction of dynamics with pixel‐level measurements

AbstractAlthough previous studies have made some clear leap in learning latent dynamics from high‐dimensional representations, the performances in terms of accuracy and inference time of long‐term model prediction still need to be improved. In this study, a deep convolutional network based on the Koopman operator (CKNet) is proposed to model non‐linear systems with pixel‐level measurements for long‐term prediction. CKNet adopts an autoencoder network architecture, consisting of an encoder to generate latent states and a linear dynamical model (i.e., the Koopman operator) which evolves in the latent state space spanned by the encoder. The decoder is used to recover images from latent states. According to a multi‐step ahead prediction loss function, the system matrices for approximating the Koopman operator are trained synchronously with the autoencoder in a mini‐batch manner. In this manner, gradients can be synchronously transmitted to both the system matrices and the autoencoder to help the encoder self‐adaptively tune the latent state space in the training process, and the resulting model is time‐invariant in the latent space. Therefore, the proposed CKNet has the advantages of less inference time and high accuracy for long‐term prediction. Experiments are performed on OpenAI Gym and Mujoco environments, including two and four non‐linear forced dynamical systems with continuous action spaces. The experimental results show that CKNet has strong long‐term prediction capabilities with sufficient precision.

Cooperative multi-agent target searching: a deep reinforcement learning approach based on parallel hindsight experience replay

Multi-agent multi-target search strategies can be utilized in complex scenarios such as post-disaster search and rescue by unmanned aerial vehicles. To solve the problem of fixed target and trajectory, the current multi-agent multi-target search strategies are mainly based on deep reinforcement learning (DRL). However, the training of agents by the DRL tend to be brittle due to their sensitivity to the training environment, which makes the strategies learned by the agents fall into local optima frequently, resulting in poor system robustness. Additionally, sparse rewards in DRL will lead to the problems such as difficulty in system convergence and low utilization efficiency of the sampled data. To address the problem that the robustness of the agents is weakened and the sparse rewards exist in the multi-objective search environment, we propose a MiniMax Multi-agent Deep Deterministic Policy Gradient based on the Parallel Hindsight Experience Replay (PHER-M3DDPG) algorithm, which adopts the framework of centralized training and decentralized execution in continuous action space. To enhance the system robustness, the PHER-M3DDPG algorithm employs a minimax learning architecture, which adaptively adjusts the learning strategy of agents by involving adversarial disturbances. In addition, to solve the sparse rewards problem, the PHER-M3DDPG algorithm adopts a parallel hindsight experience replay mechanism to increase the efficiency of data utilization by involving virtual learning targets and batch processing of the sampled data. Simulation results show that the PHER-M3DDPG algorithm outperforms the existing algorithms in terms of convergence speed and the task completion time in a multi-target search environment.

On the sample complexity of actor-critic method for reinforcement learning with function approximation

Reinforcement learning, mathematically described by Markov Decision Problems, may be approached either through dynamic programming or policy search. Actor-critic algorithms combine the merits of both approaches by alternating between steps to estimate the value function and policy gradient updates. Due to the fact that the updates exhibit correlated noise and biased gradient updates, only the asymptotic behavior of actor-critic is known by connecting its behavior to dynamical systems. This work puts forth a new variant of actor-critic that employs Monte Carlo rollouts during the policy search updates, which results in controllable bias that depends on the number of critic evaluations. As a result, we are able to provide for the first time the convergence rate of actor-critic algorithms when the policy search step employs policy gradient, agnostic to the choice of policy evaluation technique. In particular, we establish conditions under which the sample complexity is comparable to stochastic gradient method for non-convex problems or slower as a result of the critic estimation error, which is the main complexity bottleneck. These results hold in continuous state and action spaces with linear function approximation for the value function. We then specialize these conceptual results to the case where the critic is estimated by Temporal Difference, Gradient Temporal Difference, and Accelerated Gradient Temporal Difference. These learning rates are then corroborated on a navigation problem involving an obstacle and the pendulum problem which provide insight into the interplay between optimization and generalization in reinforcement learning.

Intelligent resource allocation for transmission security on IRS-assisted spectrum sharing systems with OFDM

In this paper, intelligent reflecting surface (IRS) technology is employed to enhance physical layer security (PLS) for spectrum sharing communication systems with orthogonal frequency division multiplexing (OFDM). Aiming to improve the secondary users’ secrecy rates, a design problem for jointly optimizing the transmission beamforming of secondary base station (SBS), the IRS’s reflecting coefficient and the channel allocation is formulated under the constraints of the requirements of minimum data rates of primary users and the interference between users. As the scenario is highly complex, it is quite challenging to address the non-convexity of the optimization problem. Thus, a deep reinforcement learning (DRL) based approach is taken into consideration. Specifically, we use dueling double deep Q networks (D3QN) and soft Actor–Critic (SAC) to solve the discrete and continuous action space optimization problems, respectively, taking full advantage of the maximum entropy RL algorithm to explore all possible optimal paths. Finally, simulation results show that our proposed approach has a great improvement in security transmission rate compared with the scheme without IRS and OFDM, and our proposed D3QN-SAC approach is more effective than other approaches in terms of maximum security transmission rate.

Optimization of oxygen system scheduling in hybrid action space based on deep reinforcement learning

The simultaneous consideration of discrete and continuous variables including equipment start-up, shutdown, oxygen output and dissipation, is required in the optimal problem of oxygen system scheduling in steel enterprises. To solve this problem involving hybrid variables, a hybrid actor-critic (HAC) algorithm is proposed in this paper. The proposed algorithm subdivides the action space into a discrete and continuous action space and evaluates the policy through the improved Q function. Besides, the correlation matrix is constructed to address the correspondence between hybrid actions. As a result, the generation of spurious gradients that lead to suboptimal action selection is prevented. To verify the effectiveness of the proposed approach, experiments with multiple groups employing the real data from steel enterprises are carried out. The results demonstrate that such a practice-based solution successfully resolves the oxygen scheduling problem and simultaneously improves the reward and algorithm convergence speed.

Autonomous collision avoidance system in a multi-ship environment based on proximal policy optimization method

To address the challenge of local path planning in a complex, dynamic environment with multiple ships, we built an autonomous collision avoidance system based on deep reinforcement learning in open water. The system switches between path-tracking and collision avoidance modes in real time, during which any collision danger is perceived through encounter identification and risk calculation. Subsequently, the proximal policy optimization method is applied to design the state space, action space, and reward functions. Decision-making offers advantages in terms of continuous action space, perception of global navigation information, compliance with the International Regulations for Preventing Collisions at Sea (COLREGs), and smooth maneuverings. Simulations are performed under multi-ship encounter conditions, considering the ship hydrodynamic model, environmental disturbance force model, COLREGs, and good seamanship. The results indicate the effectiveness and ability of the proposed system to take appropriate action when faced with dangerous situations under rule constraints and diverse scenarios.

Using Distributed Reinforcement Learning for Resource Orchestration in a Network Slicing Scenario

The Network Slicing (NS) paradigm enables the partition of physical and virtual resources among multiple logical networks, possibly managed by different tenants. In such a scenario, network resources need to be dynamically allocated according to the slice requirements. In this paper, we attack the above problem by exploiting a Deep Reinforcement Learning approach. Our framework is based on a distributed architecture, where multiple agents cooperate towards a common goal. The agent training is carried out following the Advantage Actor Critic algorithm, which permits to handle continuous action spaces. By means of extensive simulations, we show that our approach yields better performance than both a static allocation of system resources and an efficient empirical strategy. At the same time, the proposed system ensures high adaptability to different scenarios without the need for additional training.

Toward robust and scalable deep spiking reinforcement learning.

Deep reinforcement learning (DRL) combines reinforcement learning algorithms with deep neural networks (DNNs). Spiking neural networks (SNNs) have been shown to be a biologically plausible and energy efficient alternative to DNNs. Since the introduction of surrogate gradient approaches that allowed to overcome the discontinuity in the spike function, SNNs can now be trained with the backpropagation through time (BPTT) algorithm. While largely explored on supervised learning problems, little work has been done on investigating the use of SNNs as function approximators in DRL. Here we show how SNNs can be applied to different DRL algorithms like Deep Q-Network (DQN) and Twin-Delayed Deep Deteministic Policy Gradient (TD3) for discrete and continuous action space environments, respectively. We found that SNNs are sensitive to the additional hyperparameters introduced by spiking neuron models like current and voltage decay factors, firing thresholds, and that extensive hyperparameter tuning is inevitable. However, we show that increasing the simulation time of SNNs, as well as applying a two-neuron encoding to the input observations helps reduce the sensitivity to the membrane parameters. Furthermore, we show that randomizing the membrane parameters, instead of selecting uniform values for all neurons, has stabilizing effects on the training. We conclude that SNNs can be utilized for learning complex continuous control problems with state-of-the-art DRL algorithms. While the training complexity increases, the resulting SNNs can be directly executed on neuromorphic processors and potentially benefit from their high energy efficiency.

Maximum Entropy Exploration in Contextual Bandits with Neural Networks and Energy Based Models.

Contextual bandits can solve a huge range of real-world problems. However, current popular algorithms to solve them either rely on linear models or unreliable uncertainty estimation in non-linear models, which are required to deal with the exploration-exploitation trade-off. Inspired by theories of human cognition, we introduce novel techniques that use maximum entropy exploration, relying on neural networks to find optimal policies in settings with both continuous and discrete action spaces. We present two classes of models, one with neural networks as reward estimators, and the other with energy based models, which model the probability of obtaining an optimal reward given an action. We evaluate the performance of these models in static and dynamic contextual bandit simulation environments. We show that both techniques outperform standard baseline algorithms, such as NN HMC, NN Discrete, Upper Confidence Bound, and Thompson Sampling, where energy based models have the best overall performance. This provides practitioners with new techniques that perform well in static and dynamic settings, and are particularly well suited to non-linear scenarios with continuous action spaces.

Multi-objective optimal scheduling of charging stations based on deep reinforcement learning

With the green-oriented transition of energy, electric vehicles (EVs) are being developed rapidly to replace fuel vehicles. In the face of large-scale EV access to the grid, real-time and effective charging management has become a key problem. Considering the charging characteristics of different EVs, we propose a real-time scheduling framework for charging stations with an electric vehicle aggregator (EVA) as the decision-making body. However, with multiple optimization objectives, it is challenging to formulate a real-time strategy to ensure each participant’s interests. Moreover, the uncertainty of renewable energy generation and user demand makes it difficult to establish the optimization model. In this paper, we model charging scheduling as a Markov decision process (MDP) based on deep reinforcement learning (DRL) to avoid the afore-mentioned problems. With a continuous action space, the MDP model is solved by the twin delayed deep deterministic policy gradient algorithm (TD3). While ensuring the maximum benefit of the EVA, we also ensure minimal fluctuation in the microgrid exchange power. To verify the effectiveness of the proposed method, we set up two comparative experiments, using the disorder charging method and deep deterministic policy gradient (DDPG) method, respectively. The results show that the strategy obtained by TD3 is optimal, which can reduce power purchase cost by 10.9% and reduce power fluctuations by 69.4%.

A deep reinforcement learning-based bidding strategy for participants in a peer-to-peer energy trading scenario

An efficient energy trading strategy is proven to have a vital role in reducing participants’ payment in the energy trading process of the power grid, which can greatly improve the operation efficiency of the power grid and the willingness of participants to take part in the energy trading. Nevertheless, with the increasing number of participants taking part in the energy trading, the stability and efficiency of the energy trading system are exposed to an extreme challenge. To address this issue, an actor-critic-based bidding strategy for energy trading participants is proposed in this paper. Specifically, we model the bidding strategy with sequential decision-making characteristics as a Markov decision process, which treats three elements, namely, total supply, total demand, and participants’ individual supply or demand, as the state and regards bidding price and volume as the action. In order to address the problem that the existing value-based reinforcement learning bidding strategy cannot be applied to the continuous action space environment, we propose an actor–critic architecture, which endows the actor the ability of learning the action execution and utilizes the critic to evaluate the long-term rewards conditioned by the current state–action pairs. Simulation results in energy trading scenarios with different numbers of participants indicate that the proposed method will obtain a higher cumulative reward than the traditional greedy method.

Multiplayer reach-avoid differential games with simple motions: A review

This paper reviews the recent works on multiplayer reach-avoid (M-RA) differential games between two adversarial teams in a game region which is split into a goal region and a play region. The pursuit team aims to protect the goal region from the evasion team by cooperatively capturing the evaders which start from the play region and strive to enter the goal region. We provide a selective overview of algorithms and theoretical results for multiplayer reach-avoid differential games. Specifically, we focus on point mass holonomic players that can move freely in the game region and have simple motions as Rufus Isaacs states. We describe how the challenges due to high-dimensional continuous joint action and state spaces, as well as complex cooperations and competitions among players, can be properly resolved by a combination of qualitative and quantitative analysis of small-scale games and optimal task allocation. We finally point out the limitations of the current works and identify fruitful future research directions on theoretical studies of multiplayer reach-avoid differential games.

Adaptive Data Collection and Offloading in Multi-UAV-Assisted Maritime IoT Systems: A Deep Reinforcement Learning Approach

This paper studies the integration of data collection and offloading for maritime Internet of Things (IoT) systems with multiple unmanned aerial vehicles (UAVs). In the considered multi-UAV maritime IoT system, the UAVs act as the aerial base stations to complete the missions of data collection from buoys and data offloading to the offshore base station (OBS). In this case, the UAVs need to adaptively select the mission mode between data collection and data offloading according to the network resources and mission requirements. In this paper, we aimed to minimize the completion time of data collection and offloading missions for all UAVs by jointly optimizing the UAV trajectories, mission mode selection, transmit power of buoys, and association relationships between the UAVs and buoy/OBS. In order to solve the mixed-integer non-convex minimization problem, we first designed a multi-agent deep reinforcement learning algorithm based on a hybrid discrete and continuous action space to preliminarily obtain the UAV trajectories, mission mode selection, and the transmit power of buoys. Then, we propose an algorithm based on the stable marriage problem to determine the buoy–UAV and UAV–OBS association relationships. Finally, the simulation results show that the proposed algorithms can effectively shorten the total mission completion time of data collection and offloading for the multi-UAV-assisted maritime IoT system.

Computational Performance of Deep Reinforcement Learning to Find Nash Equilibria

We test the performance of deep deterministic policy gradient—a deep reinforcement learning algorithm, able to handle continuous state and action spaces—to find Nash equilibria in a setting where firms compete in offer prices through a uniform price auction. These algorithms are typically considered “model-free” although a large set of parameters is utilized by the algorithm. These parameters may include learning rates, memory buffers, state space dimensioning, normalizations, or noise decay rates, and the purpose of this work is to systematically test the effect of these parameter configurations on convergence to the analytically derived Bertrand equilibrium. We find parameter choices that can reach convergence rates of up to 99%. We show that the algorithm also converges in more complex settings with multiple players and different cost structures. Its reliable convergence may make the method a useful tool to studying strategic behavior of firms even in more complex settings.

Frequency regulation of multi-microgrid with shared energy storage based on deep reinforcement learning

• A frequency regulation model for microgrid with share energy storage is established. • A DRL-based economic frequency regulation method is proposed. • Performance and operating cost of frequency regulation are considered together. • Multiple frequency regulation methods are compared and analyzed. • Results show that the proposed method has obvious advantages in integrated benefit. The microgrid is one of the fundamental ways to consume renewable energy, and the safety and economy of its frequency regulation are widely concerned and studied. For the microgrid with shared energy storage, a new frequency regulation method based on deep reinforcement learning (DRL) is proposed to cope with the uncertainty of source load, which considers both frequency performance and the operational economy of the microgrid. Firstly, a frequency regulation model for the microgrid is developed by sharing the frequency regulation potential of energy consumers. Secondly, a command allocation model for smart generation control (SGC) based on the integrated benefit is proposed, where frequency safety and economy are combined. Then, a DRL framework is designed, and the twin delayed deep deterministic policy gradient algorithm is used to implement the SGC of the microgrid in the continuous action space. Finally, the effectiveness of the proposed frequency regulation method is demonstrated by designing comparative simulations in the isolated multi-microgrid, and the cost weight and responsiveness of energy consumers are analyzed.