Constrained Double Deep Q-learning Network for EVs Charging Scheduling with Renewable Energy

Abstract

With the rapid development of electric vehicles (EVs) and renewable energy, the charging scheduling of EVs with renewable energy becomes a hot issue. Nevertheless, as both the renewable energy and charging demand of EVs are hard to predict, it is difficult to get the optimal charging strategy of the EVs charging problem with renewable energy. In this paper, we consider a multi-EVs charging problem with a discontinuous charging process, where the charging decision for each EV is constrained. We model the problem as Markov decision processes (MDPs) and propose a novel Deep Reinforcement Learning (DRL) method called Constrained Double Deep Q-learning Network (CDDQN) to solve the MDP problem with large state space and decision constraints. Compared with the most of existing DRL methods, the CDDQN method embeds an action constraint model to a double deep Q-learning network (DDQN), which decreases the error of Q-value estimation and improves the accuracy of the charging policy by generating more effective training data. We conduct experiments on a simulation study and compare the proposed method with other DRL methods and experience charging policies. The quantitative results show that the proposed method achieves superior performance on getting the charging policy and outperforms the other methods in getting the minimum charging cost obviously.