Reinforcement Learning Based Energy Management Research Articles

A Data-Driven Reinforcement Learning Based Energy Management Strategy via Bridging Offline Initialization and Online Fine-Tuning for a Hybrid Electric Vehicle

A Data-Driven Reinforcement Learning Based Energy Management Strategy via Bridging Offline Initialization and Online Fine-Tuning for a Hybrid Electric Vehicle

Deep reinforcement learning based energy management of a hybrid electricity-heat-hydrogen energy system with demand response

Hybrid electricity-heat-hydrogen energy system with demand response (DR) is promising in enhancing flexibility and energy efficiency. However, the multi-energy coupling and source-load uncertainties makes it challenging to efficiently schedule energy flows of electricity generation, storage and DR. To this end, this paper proposes a continues deep reinforcement learning algorithm, specifically the deep deterministic policy gradient (DDPG), for the energy management optimization. Different Markov decision processes are firstly employed to analyze and compare two kinds of incentive-based electro-thermal DR contracts, i.e., load curtailment and load shifting. Simulation results exemplify the superiority of the proposed DDPG-based scheduling incorporating electro-thermal DR in terms of economy and sustainability, leading to a 16.02 % reduction in scheduling costs for contract load curtailment and an 8.52 % reduction for contract load shifting when compared to that without DR consideration. Furthermore, the robustness of DDPG-based scheduling is verified under 60 random source-load scenarios compared with different algorithms. Compared to results obtained by DDPG, DDPG-LC and DDPG-LS reduce the mean cost by 22.15 % and 12.84 %, while their error from the theoretical optimum is only around 5 %. The results demonstrate that the approximate optimality and rapid decision-making illustrate DDPG's efficient real-time scheduling capability, thereby enhancing the system's adaptability to uncertain environments.

Deep Reinforcement Learning based Energy Management for Heavy Duty HEV considering Discrete-Continuous Hybrid Action Space

Deep Reinforcement Learning based Energy Management for Heavy Duty HEV considering Discrete-Continuous Hybrid Action Space

A comparative study of deep reinforcement learning based energy management strategy for hybrid electric vehicle

Energy management strategies (EMSs) are essential for hybrid electric vehicles (HEVs), as they can further exploit the potential of HEVs to save energy and reduce emissions. Research on deep reinforcement learning (DRL)-based EMSs is developing rapidly. However, most studies have ignored the impact of uniform test benchmarks on the performance of DRL-based EMS and focus too much on fuel economy improvement resulting in a single optimization objective. In this study, four DRL-based EMSs are designed for HEVs with a multi-objective optimization reward function that considers battery health furtherly. The optimal learning rates and weight coefficients of the four EMSs are determined first. Based on this, the monetary cost, fuel cost, and battery health of each EMS are intensively studied under nine driving cycles. The EMSs perform better in high-speed conditions and worse in suburban conditions are initially concluded. A comparative analysis under unlearned mixed driving cycles validates this conclusion and shows that the SAC-based EMS achieves a fuel consumption of 4.218L per 100 km and 99.96 % battery health, which are the lowest of the four EMSs. This paper can provide a theoretical basis for the parametric and driving cycle study of DRL-based EMSs.

Deep reinforcement learning based energy management strategy for range extend fuel cell hybrid electric vehicle

To meet the power and long-range driving requirements of the vehicle, this paper presents a dual mode operation scheme for a range extend fuel cell hybrid vehicle for the first time, with an in-depth study of the pure electric mode and the range extend mode. The deep deterministic policy gradient algorithm is a well-known deep reinforcement learning algorithm that can solve complex nonlinear problems. To achieve the optimal power distribution among energy sources in the two modes, a dual deep deterministic policy gradient algorithm framework is proposed for the first time in this paper. In addition, a pervious action guidance mechanism is proposed to enable networks to approximate the action value function more efficiently in training. The training results show that the adopted previous action guidance mechanism helps to improve the learning convergence and exploration ability. The validation results show that the proposed strategy improves the operating economy by about 30% compared to the rule-based strategy, reduces the average fuel cell output fluctuation to less than 100 W, and reduces the fuel cell lifetime loss greatly. It is hoped that the proposed new structure, patterns, and energy management strategy will provide more ideas for scholars in future research.

Hierarchical reinforcement learning based energy management strategy of plug-in hybrid electric vehicle for ecological car-following process

The economy-oriented automated hybrid eclectic vehicles (HEV) provide great potential to save energy by optimizing both driving behaviors and power distribution. Recent advances in the ecological car following issue of HEV focus on fusing adaptive cruise control (ACC) and energy management system (EMS) by collaborative optimization. However, series control frameworks ACC+EMS breaks the internal coupling relation between motion control and energy distribution, leading to the natural limitation of its optimization. On the opposite, integrated ACC-EMS promises energy-saving improvement but brings complex optimization problems with multi-scale objectives and large exploration space. The huge computation load restricts the online application of ACC-EMS. To address these problems, a hierarchical reinforcement learning based ACC-EMS strategy is proposed with a hierarchical policy and non-hierarchical execution. The upper layer learns to plan state-of-charge and time-headway trajectories, while the low layer policy learns to achieve the expected goals by outputting control variables executed by the host vehicle. The proposed ACC-EMS strategy were self-learning by interaction in car-following scenario constructed with GPS data on I-880 highway. Comprehensive simulations show the proposed strategy has significantly improved the training speed and stability, compared to the offline global optimum, achieving the energy consumption difference of less than 3% and computational load of less than 600 times.

A comparative study of 13 deep reinforcement learning based energy management methods for a hybrid electric vehicle

Energy management strategy (EMS) has a huge impact on the energy efficiency of hybrid electric vehicles (HEVs). Recently, fast-growing number of studies have applied different deep reinforcement learning (DRL) based EMS for HEVs. However, a unified performance review benchmark is lacking for most popular DRL algorithms. In this study, 13 popular DRL algorithms are applied as HEV EMSs. The reward performance, computation cost, and learning convergence of different DRL algorithms are discussed. In addition, HEV environments are modified to fit both discrete and continuous action spaces. The results show that the stability of agent during the learning process of continuous action space is more stable than discrete action space. In the continuous action space, SAC has the highest reward, and PPO has the lowest time cost. In discrete action space, DQN has the lowest time cost, and FQF has the highest reward. The comparison among SAC, FQF, rule-based, and equivalent consumption minimization strategies (ECMS) shows that DRL EMSs run the engine more efficiently, thus saving fuel consumption. The fuel consumption of FQF is 10.26% and 5.34% less than Rule-based and ECMS, respectively. The contribution of this paper will speed up the application of DRL algorithms in the HEV EMS application.

An Assessment of Multistage Reward Function Design for Deep Reinforcement Learning-Based Microgrid Energy Management

Reinforcement learning based energy management strategy has been an active research subject in the past few years. Different from the baseline reward function (BRF), the work proposes and investigates a multi-stage reward mechanism (MSRM) that scores the agent’s step and final performance during training and returns it to the agent in real time as a reward. MSRM will also improve the agent’s training through expert intervention which aims to prevent the agent from being trapped in sub-optimal strategies. The energy management performance considered by MSRM-based algorithm includes the energy balance, economic cost, and reliability. The reward function is assessed in conjunction with two deep reinforcement learning algorithms: double deep Q-learning network (DDQN) and policy gradient (PG). Upon benchmarking with BRF, the numerical simulation shows that MSRM tends to improve the convergence characteristic, reduce the explained variance, and reduce the tendency of the agent being trapped in suboptimal strategies. In addition, the methods have been assessed with MPC-based energy management strategies in terms of relative cost, self-balancing rate, and computational time. The assessment concludes that, in the given context, PG-MSRM has the best overall performance.

Deep reinforcement learning based energy management strategy for fuel cell/battery/supercapacitor powered electric vehicle

Vehicles using a single fuel cell as a power source often have problems such as slow response and inability to recover braking energy. Therefore, the current automobile market is mainly dominated by fuel cell hybrid vehicles. In this study, the fuel cell hybrid commercial vehicle is taken as the research object, and a fuel cell/battery/supercapacitor energy topology is proposed, and an energy management strategy based on a double-delay deep deterministic policy gradient is designed for this topological structure. This strategy takes fuel cell hydrogen consumption, fuel cell life loss, and battery life loss as the optimization goals, in which supercapacitors play the role of coordinating the power output of the fuel cell and the battery, providing more optimization ranges for the optimization of fuel cells and batteries. Compared with the deep deterministic policy gradient strategy (DDPG) and the nonlinear programming algorithm strategy, this strategy has reduced hydrogen consumption level, fuel cell loss level, and battery loss level, which greatly improves the economy and service life of the power system. The proposed EMS is based on the TD3 algorithm in deep reinforcement learning, and simultaneously optimizes a number of indicators, which is beneficial to prolong the service life of the power system.

A multi-agent deep reinforcement learning based energy management for behind-the-meter resources

The future communities are becoming more and more electrically connected via increased penetrations of behind-the-meter (BTM) resources, specifically, electric vehicles (EVs), smart buildings (SBs), and distributed renewables. The electricity infrastructure is thus seeing increased challenges in its reliable, secure, and economic operation and control with increased and hard to predict demands (due to EV charging and demand management of SBs), fluctuating generation from renewables, as well as their plug-N-play dynamics. Reinforcement learning has been extensively used to enable network entities to obtain optimal policies. The recent development of deep learning has enabled deep reinforcement learning (DRL) to drive optimal policies for sophisticated and capable agents, which can outperform conventional rule-based operation policies in applications such as games, natural language processing, and biology. Furthermore, DRL has shown promising results in many resource management tasks. Numerous studies have been conducted on the application of single-agent DRL to energy management. In this paper, a fully distributed energy management framework based on multi-agent deep reinforcement learning (MADRL) is proposed to optimize the BTM resource operations and improve essential service delivery to community residents.

Hierarchical Q-learning network for online simultaneous optimization of energy efficiency and battery life of the battery/ultracapacitor electric vehicle

Reinforcement learning has been gaining attention in energy management of hybrid power systems for its low computation cost and great energy saving performance. However, the potential of reinforcement learning (RL) has not been fully explored in electric vehicle (EV) applications because most studies on RL only focused on single design targets. This paper studied on online optimization of the supervisory control system of an EV (powered by battery and ultracapacitor) with two design targets, maximizing energy efficiency and battery life. Based on a widely used reinforcement learning method, Q-learning, a hierarchical learning network is proposed. Within the hierarchical Q-learning network, two independent Q tables, Q1 and Q2, are allocated in two control layers. In addition to the baseline power-split layer, which determines the power split ratio between battery and ultracapacitor based on the knowledge stored in Q1, an upper layer is developed to trigger the engagement of the ultracapacitor based on Q2. In the learning process, Q1 and Q2 are updated during the real driving using the measured signals of states, actions, and rewards. The hierarchical Q-learning network is developed and evaluated following a full propulsion system model. By introducing the single-layer Q-learning based method and the rule-based method as two baselines, performance of the EV with the three control methods (i.e., two baseline and one proposed) are simulated under different driving cycles. The results show that the addition of an ultracapacitor in the electric vehicle reduces the battery capacity loss by 12%. The proposed hierarchical Q-learning network is shown superior to the two baseline methods by reducing 8% battery capacity loss. The vehicle range is slightly extended along with the battery life extension. Moreover, the proposed strategy is validated by considering different driving cycle and measurement noise. The proposed hierarchical strategy can be adapted and applied to reinforcement learning based energy management in different hybrid power systems.

A Comparative Study of 13 Deep Reinforcement Learning Based Energy Management Methods for a Hybrid Electric Vehicle

A Comparative Study of 13 Deep Reinforcement Learning Based Energy Management Methods for a Hybrid Electric Vehicle

Deep reinforcement learning-based energy management strategy for hybrid electric vehicles

Deep reinforcement learning-based energy management strategy for hybrid electric vehicles

Deep reinforcement learning based energy management strategy of fuel cell hybrid railway vehicles considering fuel cell aging

In the rail transportation industry, growing energy and environmental awareness requires the use of alternatives to combustion engines. These include hybrid electrically driven railway vehicles powered by fuel cells and batteries. The cost of hydrogen consumption and the lifetime of fuel cells are currently the main challenges that need to be addressed before widespread deployment of fuel cell railway vehicles can be realized. With this in mind, this work focuses on the energy management system with emphasis on optimizing the energy distribution to reduce the overall operational cost. The presented energy management strategy (EMS) aims at minimizing hydrogen consumption and fuel cell aging costs while achieving a favorable balance between battery charging and discharging. In order to take fuel cell aging into account in energy management and mitigate fuel cell aging trough power distribution, an online fuel cell aging estimation model based on four operation modes is introduced and applied. Moreover, the advanced deep reinforcement learning method Twin Delayed Deep Deterministic Policy Gradient (TD3) is used to obtain a promising EMS. To improve the adaptability of the strategy, a stochastic training environment, which is based on real measured speed profiles considering passenger numbers is used for training. Assuming different environmental and passenger transport volumes, the results confirm that the proposed TD3-EMS achieves battery charge-sustaining at low hydrogen consumption while slowing down fuel cell degradation.

Hierarchical reinforcement learning based energy management strategy for hybrid electric vehicle

As the core technology of hybrid electric vehicles (HEVs), energy management strategy directly affects the fuel consumption of vehicles. This research proposes a novel reinforcement learning (RL)-based algorithm for energy management strategy of HEVs. Hierarchical structure is used in deep Q-learning algorithm (DQL-H) to get the optimal solution of energy management. Through this new RL method, we not only solve the problem of sparse reward in training process, but also achieve the optimal power distribution. In addition, as a kind of hierarchical algorithm, DQL-H can change the way of exploration of the vehicle environment and make it more effective. The experimental results show that the proposed DQL-H method realizes better training efficiency and lower fuel consumption, compared to other RL-based ones.

A Deep Reinforcement Learning Based Energy Management Strategy for Hybrid Electric Vehicles in Connected Traffic Environment

Abstract This paper proposed a deep reinforcement learning based energy management strategy in the connected traffic environment. At the upper layer, a deep deterministic policy gradient algorithm is used to systematically integrate ego vehicle reference speed planning into the energy management strategy in the car following scenarios. At the bottom layer, a driver model and an adaptive equivalent consumption minimization strategy are implemented to conduct the optimal power split control based on the planned reference speed. As distance headway, fuel consumption, and terrain information are taken into consideration by the deep reinforcement learning controller, the proposed strategy can not only maintain a safe distance headway but also improve the energy efficiency of the entire driving cycle. Finally, the fuel consumption score of the proposed strategy is reduced by 3.5% compared with a car following strategy based on a proportional-integral controller. The simulation results show that the proposed strategy can learn a reasonable car following policy through the training process. In addition, the simulation duration for each reference speed planning step is only about 4.1 ms, which proves the proposed strategy can be applied online.

Cost-effective reinforcement learning energy management for plug-in hybrid fuel cell and battery ships

Hybrid fuel cell and battery propulsion systems have the potential to offer improved emission performance for coastal ships with access to H2 replenishment and battery charging infrastructures in ports. However, such systems could be constrained by high power source degradation and energy costs. Cost-effective energy management strategies are essential for such hybrid systems to mitigate the high costs. This article presents a Double Q reinforcement learning based energy management system for such systems to achieve near-optimal average voyage cost. The Double Q agent is trained using stochastic power profiles collected from continuous monitoring of a passenger ferry, using a plug-in hybrid fuel cell and battery propulsion system model. The energy management strategies generated by the agent were validated using another test dataset collected over a different period. The proposed methodology provides a novel approach to optimal use hybrid fuel cell and battery propulsion systems for ships. The results show that without prior knowledge of future power demands, the strategies can achieve near-optimal cost performance (96.9%) compared to those derived from using dynamic programming with the equivalent state space resolution.

Tackling SOC long-term dynamic for energy management of hybrid electric buses via adaptive policy optimization

Plug-in hybrid electric buses (PHEBs) have the potential to satisfy both the fuel efficiency and the driving-mileage under complex urban traffic conditions. However, the optimal charge and discharge management is still a pivotal challenge of energy management for the inherent uncertainty in driving conditions. The common reference state-of-charge (SOC) profile based methods are limited by the adaptiveness which restricts the economic performance of on-line energy management systems. Promisingly, reinforcement learning based energy management strategies exhibited the significant self-learning ability. However, for PHEBs, the sparse rewards by the long-term SOC shortage make the strategies easily trick into the local optimal solution. The work presented in this paper concentrates on combining battery power reduction in the form of conditional entropy into reinforcement learning based energy management strategy. The proposed method named adaptive policy optimization (APO) introduces a novel advantage function to evaluate energy-saving performance considering long-term SOC dynamic, and a Bayesian neural network based SOC shortage probability estimator is utilized to optimize the energy management strategy parameterized by a deep neural network. Several experiments in a standard driving cycle demonstrate the optimality, self-learning ability and convergence of the APO. Moreover, the adaptability and robust performance get validated over the real bus trajectories data. With the comprehensive experiments in this paper, the proposed model exhibits enhanced fuel economy and more suitable SOC planning in comparison with the existing energy management strategies. The results indicate that APO respectively outperforms the compared online strategies by 9.8% and 2.6% and reaches 98% energy-saving rate of the offline global optimum.

Deep reinforcement learning based energy management for a hybrid electric vehicle

This research proposes a reinforcement learning-based algorithm and a deep reinforcement learning-based algorithm for energy management of a series hybrid electric tracked vehicle. Firstly, the powertrain model of the series hybrid electric tracked vehicle (SHETV) is constructed, then the corresponding energy management formulation is established. Subsequently, a new variant of reinforcement learning (RL) method Dyna, namely Dyna-H, is developed by combining the heuristic planning step with the Dyna agent and is applied to energy management control for SHETV. Its rapidity and optimality are validated by comparing with DP and conventional Dyna method. Facing the problem of the “curse of dimensionality” in the reinforcement learning method, a novel deep reinforcement learning algorithm deep Q-learning (DQL) is designed for energy management control, which uses a new optimization method (AMSGrad) to update the weights of the neural network. Then the proposed deep reinforcement learning control system is trained and verified by the realistic driving condition with high-precision, and is compared with the benchmark method DP and the traditional DQL method. Results show that the proposed deep reinforcement learning method realizes faster training speed and lower fuel consumption than traditional DQL policy does, and its fuel economy quite approximates to global optimum. Furthermore, the adaptability of the proposed method is confirmed in another driving schedule.

Rule-interposing deep reinforcement learning based energy management strategy for power-split hybrid electric vehicle

The optimization and training processes of deep reinforcement learning (DRL) based energy management strategy (EMS) can be very slow and resource-intensive. In this paper, an improved energy management framework that embeds expert knowledge into deep deterministic policy gradient (DDPG) is proposed. Incorporated with the battery characteristics and the optimal brake specific fuel consumption (BSFC) curve of hybrid electric vehicles (HEVs), we are committed to solving the optimization problem of multi-objective energy management with a large space of control variables. By incorporating this prior knowledge, the proposed framework not only accelerates the learning process, but also gets a better fuel economy, thus making the energy management system relatively stable. The experimental results show that the proposed EMS outperforms the one without prior knowledge and the other state-of-art deep reinforcement learning approaches. In addition, the proposed approach can be easily generalized to other types of HEV EMSs.