Deep Deterministic Policy Gradient Method Research Articles

Federated reinforcement learning for smart building joint peer-to-peer energy and carbon allowance trading

The multi-energy system (MES), which is regarded as an optimum solution to a high-efficiency, green energy system and a crucial shift towards the future low-carbon energy system, has attracted great attention at the district building level. However, the current exploration of flexible MES operation has been hampered by (1) the increasing penetration of renewable energies and the complicated operation of coupling multi-energy sectors; (2) the privacy concern in the decentralization of the energy system; and (3) the lack of integration of the energy market and carbon emission trading scheme. To address the aforementioned challenges, this paper proposes a joint peer-to-peer energy and carbon allowance trading mechanism for a building community, and then models it as a multi-agent reinforcement learning (MARL) paradigm. In this setting, the flexibility of building local trading and the decarbonization of building energy management can both be fully utilized. To stabilize the training performance, an abstract critic network capturing system dynamics is introduced based on a deep deterministic policy gradient method. The technique of federated learning (FL) is also applied to speed up the training and safeguard the private information of each building in the community. Empirical results on a real-world test case evaluate its superior performance in terms of achieving both economic and environmental benefits, resulting in 5.87% and 8.02% lower total energy and environment costs than the two baseline mechanisms of peer-to-grid energy trading and peer-to-peer energy trading, respectively.

An Iterative Optimization and Learning-Based IoT System for Energy Management of Connected Buildings

Buildings account for nearly 40% of primary energy and 36% of greenhouse emissions, which is one of the main factors driving climate change. Reducing energy consumption in buildings toward zero-energy buildings is a vital pillar to ensure that future climate and energy targets are reached. However, due to the high uncertainty of building loads and customer comfort demands, and extremely nonlinear building thermal characteristics, developing an effective zero-energy building energy management (BEM) technology is facing great challenges. This article proposes a novel learning-based and iterative Internet of Things (IoT) system to address these challenges to achieve the zero-energy objective in BEM of connected buildings. First, all buildings in the IoT-based BEM system share their operation data with an aggregator. Second, the aggregator uses these historical data to train a deep reinforcement learning model based on the deep deterministic policy gradient method. The learning model generates precooling or preheating control actions to achieve zero-energy BEM for building heating ventilation and air conditioning (HVAC) systems. Third, for solving the coupling problem between HVAC systems and building internal heat gain loads, an iterative optimization algorithm is developed to integrate physics-based and learning-based models to minimize the deviation between the on-site solar photovoltaic generated energy and the actual building energy consumption by properly scheduling building loads, electric vehicle charging cycles, and the energy-storage system. Finally, the optimal load operation scheduling is generated by considering customers’ comfort requirements. All connected buildings then operate their loads based on the load operation schedule issued by the aggregator. The proposed learning-based and iterative IoT system is validated via simulation with real-world building data from the Pecan Street project.

Multiagent Deep Reinforcement Learning for Cost- and Delay-Sensitive Virtual Network Function Placement and Routing

This paper proposes an effective and novel multi-agent deep reinforcement learning (MADRL)-based method for solving the joint virtual network function (VNF) placement and routing (P&R), where multiple service requests with differentiated demands are delivered at the same time. The differentiated demands of the service requests are reflected by their delay- and cost-sensitive factors. We first construct a VNF P&R problem to jointly minimize a weighted sum of service delay and resource consumption cost, which is NP-complete. Then, the joint VNF P&R problem is decoupled into two iterative subtasks: placement subtask and routing subtask. Each subtask consists of multiple concurrent parallel sequential decision processes. By invoking the deep deterministic policy gradient method and multi-agent technique, an MADRL-P&R framework is designed to perform the two subtasks. The new <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">joint reward and internal rewards</i> mechanism is proposed to match the goals and constraints of the placement and routing subtasks. We also propose the parameter migration-based model-retraining method to deal with changing network topologies. Corroborated by experiments, the proposed MADRL-P&R framework is superior to its alternatives in terms of service cost and delay, and offers higher flexibility for personalized service demands. The parameter migration-based model-retraining method can efficiently accelerate convergence under moderate network topology changes.

Learning-Based Model Predictive Control of DC-DC Buck Converters in DC Microgrids: A Multi-Agent Deep Reinforcement Learning Approach

This paper proposes a learning-based finite control set model predictive control (FCS-MPC) to improve the performance of DC-DC buck converters interfaced with constant power loads in a DC microgrid (DC-MG). An approach based on deep reinforcement learning (DRL) is presented to address one of the ongoing challenges in FCS-MPC of the converters, i.e., optimal design of the weighting coefficients appearing in the FCS-MPC objective function for each converter. A deep deterministic policy gradient method is employed to learn the optimal weighting coefficient design policy. A Markov decision method formulates the DRL problem. The DRL agent is trained for each converter in the MG, and the weighting coefficients are obtained based on reward computation with the interactions between the MG and agent. The proposed strategy is wholly distributed, wherein agents exchange data with other agents, implying a multi-agent DRL problem. The proposed control scheme offers several advantages, including preventing the dependency of the converter control system on the operating point conditions, plug-and-play capability, and robustness against the MG uncertainties and unknown load dynamics.

A Novel Dynamic Lane-Changing Trajectory Planning Model for Automated Vehicles Based on Reinforcement Learning

Lane changing behavior has a significant impact on traffic efficiency and may lead to traffic delays or even accidents. It is important to plan a safe and efficient lane-changing trajectory that coordinates with the surrounding environment. Most conventional lane-changing models need to establish and solve constrained optimization models during the whole process, while reinforcement learning can just take the current state as input and directly output actions to vehicles. This study develops a lane-changing model using the deep deterministic policy gradient method, which can simultaneously control the lateral and longitudinal motions of the vehicle. To optimize its performance, a reward function is properly designed by combining safety, efficiency, gap, headway, and comfort features. To avoid collisions, a safety modification model is developed to check and correct acceleration at every time step. The driving trajectory data of 1169 lane-changing scenarios extracted from the Next Generation Simulation (NGSIM) dataset are used to train and test the model. The proposed model can quickly converge in training phase. Testing results show it can complete safe and efficient lane changing in different lane-changing scenarios with both shorter time headway and lane-changing duration than human drivers. Compared with the conventional dynamic lane-changing trajectory planning model, our model can reduce collision risk. It is also evaluated in automated and nonautomated mixed traffic in SUMO. Simulation results show that the proposed model also has a positive effect on the average speed of overall traffic flow.

Data-driven online interactive bidding strategy for demand response

Demand response (DR), as one of the important energy resources in the future’s grid, provides the services of peak shaving, enhancing the efficiency of renewable energy utilization with a short response period, and low cost. Various categories of DR are established, e.g. automated DR, incentive DR, emergency DR, and demand bidding. However, with the practical issue of the unawareness of residential and commercial consumers’ utility models, the researches about demand bidding aggregator involved in the electricity market are just at the beginning stage. For this issue, the bidding price and bidding quantity are two required decision variables while considering the uncertainties due to the market and participants. In this paper, we determine the bidding and purchasing strategy simultaneously employing the smart meter data and functions. A two-agent deep deterministic policy gradient method is developed to optimize the decisions through learning historical bidding experiences. The online learning further utilizes the daily newest bidding experience attained to ensure trend tracing and self-adaptation. Two environment simulators are adopted for testifying the robustness of the model. The results prove that when facing diverse situations the proposed model can earn the optimal profit via off/online learning the bidding rules and robustly making the proper bid.

Deep Reinforcement Learning From Demonstrations to Assist Service Restoration in Islanded Microgrids

Microgrids can be operated in island mode during utility grid outages to support service restoration and improve system resilience. To schedule and dispatch distributed energy resources (DERs) in an islanded microgrid, conventional model-based methods rely on accurate distribution network models and lack generalization and adaptability. Data-driven methods are promising for DER coordination but face practical challenges such as potential hazards to microgrids during online training and insufficient online training opportunities due to low outage rates. This paper presents a novel two-stage learning framework to identify an optimal restoration strategy. The proposed framework builds on the deep deterministic policy gradient from demonstrations, which is a dataset that contains a trajectory of states and the associated expert actions. At the pre-training stage, imitation learning is applied to equip the control agent with expert experiences to guarantee acceptable initial performance. At the online training stage, action clipping, reward shaping, and expert demonstrations are leveraged to ensure safe exploration while accelerating the training process. The proposed method is illustrated using the IEEE 123-node system and compared with a representative model-based method and the standard deep deterministic policy gradient method to prove solution accuracy and demonstrate increased computational efficiency.

UAV Path Planning Based on Multicritic-Delayed Deep Deterministic Policy Gradient

Deep deterministic policy gradient (DDPG) algorithm is a reinforcement learning method, which has been widely used in UAV path planning. However, the critic network of DDPG is frequently updated in the training process. It leads to an inevitable overestimation problem and increases the training computational complexity. Therefore, this paper presents a multicritic-delayed DDPG method for solving the UAV path planning. It uses multicritic networks and delayed learning methods to reduce the overestimation problem of DDPG and adds noise to improve the robustness in the real environment. Moreover, a UAV mission platform is built to train and evaluate the effectiveness and robustness of the proposed method. Simulation results show that the proposed algorithm has a higher convergence speed, a better convergence effect, and stability. It indicates that UAV can learn more knowledge from the complex environment.

Risk-sensitive policies for portfolio management

The decision-making on portfolio investment is fundamental in the financial market, but getting the optimal strategy is challenging due to high uncertainty and massive noise in the market. Deep Deterministic Policy Gradient (DDPG), proposed by Lillicrap et al. (2015), is a deep Reinforcement Learning (RL) algorithm that made remarkable achievements in the financial perspective. Although the applications of RL in financial trading are well-developed, it is surprising that most of the literature ignores the possible risk of rare occurrences of catastrophic events and the effect of the worst-case scenarios on trading decisions. In this paper, we first develop a novel deep RL algorithm, called Hierarchical DDPG, that combines the classical DDPG algorithm and the Hierarchical RL structure to control the risk of portfolio investment. Second, we adapt the distributional DDPG method for portfolio management problems, which aims to maximize the α-percentile expectation based on the distribution of future returns. A real world dataset is used to validate the performance of our proposed models. The experimental results show that our proposed models outperform the market and classical DDPG, and moreover, both approaches provide effective methods of constructing a risk-sensitive policy to protect investors from suffering a huge loss.

A Hidden Attack Sequences Detection Method Based on Dynamic Reward Deep Deterministic Policy Gradient

Attacker identification from network traffic is a common practice of cyberspace security management. However, network administrators cannot cover all security equipment due to the cyberspace management cost constraints, giving attackers the chance to escape from the surveillance of network security administrators by legitimate actions and to perform the attack in both physical domain and digital domain. Therefore, we proposed a hidden attack sequence detection method based on reinforcement learning to deal with the challenge through modeling the network administrators as an intelligent agent that learns their action policy from the interaction with the cyberspace environment. Following Deep Deterministic Policy Gradient (DDPG), the intelligent agent can not only discover the hidden attackers hiding in the legitimate action sequences but also reduce the cyberspace management cost. Furthermore, a dynamic reward DDPG method was proposed to improve defense performance, which set dynamic reward depending on the hidden attack sequences steps and agent’s check steps, compared to the fixed reward in common methods. Meanwhile, the method was verified in a simulated experimental cyberspace environment. Finally, the experimental results demonstrate that there are hidden attack sequences in cyberspace, and the proposed method can discover the hidden attack sequences. The dynamic reward DDPG shows superior performance in detecting hidden attackers, with a detection rate of 97.46%, which can improve the ability to discover hidden attackers and reduce the 6% cyberspace management cost compared to DDPG.

Building Energy Consumption Prediction Using a Deep-Forest-Based DQN Method

When deep reinforcement learning (DRL) methods are applied in energy consumption prediction, performance is usually improved at the cost of the increasing computation time. Specifically, the deep deterministic policy gradient (DDPG) method can achieve higher prediction accuracy than deep Q-network (DQN), but it requires more computing resources and computation time. In this paper, we proposed a deep-forest-based DQN (DF–DQN) method, which can obtain higher prediction accuracy than DDPG and take less computation time than DQN. Firstly, the original action space is replaced with the shrunken action space to efficiently find the optimal action. Secondly, deep forest (DF) is introduced to map the shrunken action space to a single sub-action space. This process can determine the specific meaning of each action in the shrunken action space to ensure the convergence of DF–DQN. Thirdly, state class probabilities obtained by DF are employed to construct new states by considering the probabilistic process of shrinking the original action space. The experimental results show that the DF–DQN method with 15 state classes outperforms other methods and takes less computation time than DRL methods. MAE, MAPE, and RMSE are decreased by 5.5%, 7.3%, and 8.9% respectively, and R2 is increased by 0.3% compared to the DDPG method.

Twin-Delayed Deep Deterministic Policy Gradient for Low-Frequency Oscillation Damping Control

Due to the large scale of power systems, latency uncertainty in communications can cause severe problems in wide-area measurement systems. To resolve this issue, a significant amount of past work focuses on using emerging technology, including machine learning methods such as Q-learning, for addressing latency issues in modern controls. Although the method can deal with the stochastic characteristics of communication latency, the Q-values can be overestimated in Q-learning methods, leading to high bias. To address the overestimation bias issue, we redesign the learning structure of the deep deterministic policy gradient (DDPG). Then we develop a damping control twin-delayed deep deterministic policy gradient method to handle the damping control issue under unknown latency in the power network. The purpose is to address the damping control issue under unknown latency in the power network. This paper will create a novel reward algorithm, taking into account the machine speed deviation, the episode termination prevention, and the feedback from action space. In this way, the system optimally damps down frequency oscillations while maintaining the system’s stability and reliable operation within defined limits. The simulation results verify the proposed algorithm in various perspectives, including the latency sensitivity analysis under high renewable energy penetration and the comparison with conventional and machine learning control algorithms. The proposed method shows a fast learning curve and good control performance under varying communication latency.

A novel asynchronous deep reinforcement learning model with adaptive early forecasting method and reward incentive mechanism for short-term load forecasting

Accurate load forecasting is challenging due to the significant uncertainty of load demand. Deep reinforcement learning, which integrates the nonlinear fitting ability of deep learning with the decision-making ability of reinforcement learning, has obtained effective solutions to various optimization problems. However, no study has been reported, which used deep reinforcement learning for short-term load forecasting because of the difficulties in handling the high temporal correlation and high convergence instability. In this study, a novel asynchronous deep reinforcement learning model is proposed for short-term load forecasting by addressing the above difficulties. First, a new asynchronous deep deterministic policy gradient method is proposed to disrupt the temporal correlation of different samples to reduce the overestimation of the expected total discount reward of the agent. Further, a new adaptive early forecasting method is proposed to reduce the time cost of model training by adaptively judging the training situation of the agent. Moreover, a new reward incentive mechanism is proposed to stabilize the convergence of model training by taking into account the trend of agent actions at different time steps. The experimental results show that the proposed model achieves higher forecasting accuracy, less time cost, and more stable convergence compared with eleven baseline models.

UAV Control for Wireless Service Provisioning in Critical Demand Areas: A Deep Reinforcement Learning Approach

In this paper, we investigate the problem of wireless service provisioning through a rotary-wing UAV which can serve as an aerial base station (BS) to communicate with multiple ground terminals (GTs) in a boost demand area. Our objective is to optimize the UAV control for maximizing the UAV.s energy efficiency, in which both aerodynamic energy and communication energy are considered while ensuring the communication requirements for each GT and backhaul link between the UAV and the terrestrial BS. The mobility of the UAV and GTs lead to time-varying channel conditions that make the environment dynamic. We formulate a nonconvex optimization for controlling the UAV considering the practical angle-dependent Rician fading channels between the UAV and GTs, and between the UAV and the terrestrial BS. Traditional optimization approaches are not able to handle the dynamic environment and high complexity of the problem in real-time. We propose to use a deep reinforcement learning-based approach namely Deep Deterministic Policy Gradient (DDPG) to solve the formulated nonconvex problem of UAV control with continuous action space that takes into account the real-time of the environment including time-varying UAV-ground channel conditions, available onboard energy of the UAV, and the communication requirement of the GTs. However, the DDPG method may not achieve good performance in an unstable environment and will face a large number of hyperparameters. We extend our approach to use the Trust Region Policy Optimization (TRPO) method that can improve the performance of the UAV compared to the DDPG method in such a dynamic environment.

Photovoltaic System MPPT Evaluation Using Classical, Meta-Heuristics, and Reinforcement Learning-Based Controllers: A Comparative Study

Maximum power point tracking (MPPT) entails constraining photovoltaic (PV) modules to operate under a specified power condition. It has previously been shown that some meta-heuristic techniques often suffer from steady-state oscillations around maximum points and experience difficulty in adapting to environmental variations, such as irradiation and/or temperature. To address the aforementioned limitation, this work proposed an adaptable reinforcement learning (RL) technique based on a novel deep deterministic policy gradient (DDPG) agent and a reward function. The actor–network top layer uses a sigmoid activation function and the critic–network contains bottleneck layers with non-uniform nodal distributions as well as exponential linear unit (ELU) activation functions in some of the layers. The RL based on DDPG method was compared with Particle Swarm Optimization (PSO) and Perturb-and-Observe (P&amp;O) in order to determine the optimal duty-cycle command needed for controlling the PV modules MPPT. All the investigated systems were implemented in MATLAB/Simulink. The results show that the proposed RL technique based on DDPG agent yielded superior tracking efficiency than all the other approaches. However, as the step change in irradiation at a constant temperature increases, the RL technique based on DDPG agent shows a decrease in tracking efficiency.

Hbox {CDBTune}^{+}: An efficient deep reinforcement learning-based automatic cloud database tuning system

Configuration tuning is vital to optimize the performance of a database management system (DBMS). It becomes more tedious and urgent for cloud databases (CDB) due to diverse database instances and query workloads, which make the job of a database administrator (DBA) very difficult. Existing solutions for automatic DBMS configuration tuning have several limitations. Firstly, they adopt a pipelined learning model but cannot optimize the overall performance in an end-to-end manner. Secondly, they rely on large-scale high-quality training samples which are hard to obtain. Thirdly, existing approaches cannot recommend reasonable configurations for a large number of knobs to tune whose potential values live in such high-dimensional continuous space. Lastly, in cloud environments, existing approaches can hardly cope with the changes of hardware configurations and workloads, and have poor adaptability. To address these challenges, we design an end-to-end automatic CDB tuning system, {texttt {CDBTune}}^{+}, using deep reinforcement learning (RL). {texttt {CDBTune}}^{+} utilizes the deep deterministic policy gradient method to find the optimal configurations in a high-dimensional continuous space. {texttt {CDBTune}}^{+} adopts a trial-and-error strategy to learn knob settings with a limited number of samples to accomplish the initial training, which alleviates the necessity of collecting a massive amount of high-quality samples. {texttt {CDBTune}}^{+} adopts the reward-feedback mechanism in RL instead of traditional regression, which enables end-to-end learning and accelerates the convergence speed of our model and improves the efficiency of online tuning. Besides, we propose effective techniques to improve the training and tuning efficiency of {texttt {CDBTune}}^{+} for practical usage in a cloud environment. We conducted extensive experiments under 7 different workloads on real cloud databases to evaluate {texttt {CDBTune}}^{+}. Experimental results showed that {texttt {CDBTune}}^{+} adapts well to a new hardware environment or workload, and significantly outperformed the state-of-the-art tuning tools and DBA experts.

Deep Reinforcement Learning-Based Multi-Hop State-Aware Routing Strategy for Wireless Sensor Networks

With the development of wireless sensor network technology, the routing strategy has important significance in the Internet of Things. An efficient routing strategy is one of the fundamental technologies to ensure the correct and fast transmission of wireless sensor networks. In this paper, we study how to combine deep learning technology with routing technology to propose an efficient routing strategy to cope with network topology changes. First, we use the recurrent neural network combined with the deep deterministic policy gradient method to predict the network traffic distribution. Second, the multi-hop node state is considered as the input of a double deep Q network. Therefore, the nodes can make routing decisions according to the current state of the network. Multi-hop state-aware routing strategy based on traffic flow forecasting (MHSA-TFF) is proposed. Simulation results show that the MHSA-TFF can improve transmission delay, average routing length, and energy efficiency.

Reliability-Optimal Offloading in Low-Latency Edge Computing Networks: Analytical and Reinforcement Learning Based Designs

In this paper, we consider a multi-access edge computing (MEC) network with multiple servers. Due to the low latency constraints, the wireless data transmission/offloading is carried by finite blocklength codes. We characterize the reliability of the transmission phase in the finite blocklength regime and investigate the extreme event of queue length violation in the computation phase by applying extreme value theory. Under the assumption of perfect channel state information (CSI), we follow the obtained characterizations and provide an optimal framework design including server selection and time allocation aiming to minimize the overall error probability. Moreover, when only the outdated CSI is available, a deep reinforcement learning based design is proposed applying the deep deterministic policy gradient method. Via simulations, we validate the convexity proven in our analytical model and show the performance advantage of proposed analytical solution and learning-based solution comparing to the benchmark for perfect CSI and outdated CSI, respectively.

Path Following Control for UAV Using Deep Reinforcement Learning Approach

Unmanned aerial vehicles (UAVs) have been extensively used in civil and industrial applications due to the rapid development of the guidance, navigation and control (GNC) technologies. Especially, using deep reinforcement learning methods for motion control acquires a major progress recently, since deep [Formula: see text]-learning algorithm has been successfully applied to the continuous action domain problem. This paper proposes an improved deep deterministic policy gradient (DDPG) algorithm for path following control problem of UAV. A specific reward function is designed for minimizing the cross-track error of the path following problem. In the training phase, a double experience replay buffer (DERB) is used to increase the learning efficiency and accelerate the convergence speed. First, the model of UAV path following problem has been established. After that, the framework of DDPG algorithm is constructed. Then the state space, action space and reward function of the UAV path following algorithm are designed. DERB is proposed to accelerate the training phase. Finally, simulation results are carried out to show the effectiveness of the proposed DERB–DDPG method.

A Hybrid Tracking Control Strategy for Nonholonomic Wheeled Mobile Robot Incorporating Deep Reinforcement Learning Approach

Tracking control is an essential capability for nonholonomic wheeled mobile robots (NWMR) to achieve autonomous navigation. This paper presents a novel hybrid control strategy combined mode-based control and actor-critic based deep reinforcement learning method. Based on the Lyapunov method, a kinematics control law named given control is obtained with pose errors. Then, the tracking control problem is converted to a finite Markov decision process, in which the defined state contains current tracking errors, given control inputs and one-step errors. After training with deep deterministic policy gradient method, the action named acquired control inputs is capable of compensating the existing errors. Thus, the hybrid control strategy is obtained under velocity constraint, acceleration constraint and bounded uncertainty. A cumulative error is also defined as a criteria to evaluate tracking performance. The comparison results in simulation demonstrate that our proposed method have an obviously advantage on both tracking accuracy and training efficiency.