Deterministic Policy Gradient Research Articles

A novel approach for self-driving car in partially observable environment using life long reinforcement learning

Despite ground-breaking advancements in robotics, gaming, and other challenging domains, reinforcement learning still faces significant challenges in solving dynamic, open-world problems. Since reinforcement learning algorithms usually perform poorly when exposed to new tasks outside of their data distribution, continuous learning algorithms have drawn significant attention. In parallel with work on lifelong learning algorithms, there is a need for challenging environments, properly planned trials, and metrics to measure research success. In this context, a Deep Asynchronous Autonomous Learning System (DAALS) is proposed in this paper for training a self-driving car in a partially observable environment for real-world scenarios in a continuous state-action space. To cater to three different use cases, three different algorithms were used. To train their agents for learning and upgrading discrete state policies, DAALS used the Asynchronous Advantage Stager Reviewer (AASR) algorithm. To train its agent for continuous state spaces, DAALS also uses an Extensive Deterministic Policy Gradient (EDPG) algorithm. To train the agent in a lifelong form of learning for partially observable environments, DAALS uses a Deep Deterministic Policy Gradient Novel Lifelong Learning Algorithm (DDPGNLLA). The system offers flexibility to the user to train the agents for both discrete and continuous state-action spaces. Compared to previous models in continuous state-action spaces, Deep deterministic policy gradient lifelong learning algorithm outperforms previous models by 46.09%. Furthermore, the Deep Asynchronous Autonomous System tends to outperform all previous reinforcement learning algorithms, making our proposed approach a real-world solution. As DAALS has tested on number of different environments it provides the insights on how modern Artificial Intelligence (AI) solutions can be generalized making it one of the better solutions for AI general domain problems.

Path planning of autonomous underwater vehicle in unknown environment based on improved deep reinforcement learning

In order to improve the autonomous navigation capability of autonomous underwater vehicles (AUV), this paper applies Deep Reinforcement Learning (DRL) to realize autonomous path planning. In response to the limited detection range and difficult tracking DRL policy in three-dimensional unknown environments, this paper proposes an improved deep reinforcement learning method. Firstly, performing AUV modeling and processing obstacle detection information. Second, under a framework of reinforcement learning with the control system, an improved environmental interaction method was designed by analyzing the motion characteristics and controller performance of the AUV. Finally, based on this method, a policy network that can generate paths that satisfy obstacle avoidance and navigation is trained in combination with a twin delayed deterministic policy gradient (TD3). The simulation results show that the proposed method can achieve AUV path planning in a three-dimensional unknown environment. The proposed framework can ensure that the planned path meets the dynamics constraints and is trackable. The improved algorithm can enhance the convergence of training and the attitude stability of AUV.

Deep reinforcement learning approach with hybrid action space for mobile charging in wireless rechargeable sensor networks

Mobile charging is feasible to deal with the energy-constrained problem in wireless rechargeable sensor networks (WRSNs). The mobile chargers (MCs) are usually employed to charge the sensors sequentially according to the charging schemes. Existing studies assume that each sensor should be charged to its maximum energy capacity or to a fixed upper threshold before the next one can be charged. However, they neglect to control the charging time adaptively of each sensor according to the charging demand. Hence, in this paper, we assume that the charging time of each sensor can be controlled, and we study the joint optimization of charging sequence and charging time problem (JCSCT). Correspondingly, we propose a novel deep reinforcement learning with hybrid action space approach for JCSCT (DRLH-JCSCT), which utilizes deep q-network (DQN) to generate the charging sequence, and adopts deep deterministic policy gradient (DDPG) to determine the charging time. An attention-based encoder–decoder model is integrated in the actor network of DDPG, and a modified bi-directional gate recurrent unit network (MBGRU) is utilized as the decoder. We also design a novel reward function to evaluate the quality of the charging actions. Simulations demonstrate the improved charging performance of the proposed approach, with a longer network lifetime and fewer failed sensors compared with the existing mobile charging scheduling approaches.

Deep Reinforcement Learning Car-Following Control Based on Multivehicle Motion Prediction

Reinforcement learning (RL)–based car-following (CF) control strategies have attracted significant attention in academia, emerging as a prominent research topic in recent years. Most of these control strategies focus solely on the motion status of the immediately preceding vehicle. However, with the development of vehicle-to-vehicle (V2V) communication technologies, intelligent vehicles such as connected autonomous vehicles (CAVs) can gather information about surrounding vehicles. Therefore, this study proposes an RL-based CF control strategy that takes multivehicle scenarios into account. First, the trajectories of two preceding vehicles and one following vehicle relative to the subject vehicle (SV) are extracted from a highD dataset to construct the environment. Then the twin-delayed deep deterministic policy gradient (TD3) algorithm is implemented as the control strategy for the agent. Furthermore, a sequence-to-sequence (seq2seq) module is developed to predict the uncertain motion statuses of surrounding vehicles. Once integrated into the RL framework, this module enables the agent to account for dynamic changes in the traffic environment, enhancing its robustness. Finally, the performance of the CF control strategy is validated both in the highD dataset and in two traffic perturbation scenarios. In the highD dataset, the TD3-based prediction CF control strategy outperforms standard RL algorithms in terms of convergence speed and rewards. Its performance also surpasses that of human drivers in safety, efficiency, comfort, and fuel consumption. In traffic perturbation scenarios, the performance of the proposed CF control strategy is compared with the model predictive controller (MPC). The results show that the TD3-based prediction CF control strategy effectively mitigates undesired traffic waves caused by the perturbations from the head vehicle. Simultaneously, it maintains the desired traffic state and consistently ensures a stable and efficient traffic flow.

Optimization of news dissemination push mode by intelligent edge computing technology for deep learning

The Internet era is an era of information explosion. By 2022, the global Internet users have reached more than 4 billion, and the social media users have exceeded 3 billion. People face a lot of news content every day, and it is almost impossible to get interesting information by browsing all the news content. Under this background, personalized news recommendation technology has been widely used, but it still needs to be further optimized and improved. In order to better push the news content of interest to different readers, users' satisfaction with major news websites should be further improved. This study proposes a new recommendation algorithm based on deep learning and reinforcement learning. Firstly, the RL algorithm is introduced based on deep learning. Deep learning is excellent in processing large-scale data and complex pattern recognition, but it often faces the challenge of low sample efficiency when it comes to complex decision-making and sequential tasks. While reinforcement learning (RL) emphasizes learning optimization strategies through continuous trial and error through interactive learning with the environment. Compared with deep learning, RL is more suitable for scenes that need long-term decision-making and trial-and-error learning. By feeding back the reward signal of the action, the system can better adapt to the unknown environment and complex tasks, which makes up for the relative shortcomings of deep learning in these aspects. A scenario is applied to an action to solve the sequential decision problem in the news dissemination process. In order to enable the news recommendation system to consider the dynamic changes in users' interest in news content, the Deep Deterministic Policy Gradient algorithm is applied to the news recommendation scenario. Opposing learning complements and combines Deep Q-network with the strategic network. On the basis of fully summarizing and thinking, this paper puts forward the mode of intelligent news dissemination and push. The push process of news communication information based on edge computing technology is proposed. Finally, based on Area Under Curve a Q-Leaning Area Under Curve for RL models is proposed. This indicator can measure the strengths and weaknesses of RL models efficiently and facilitates comparing models and evaluating offline experiments. The results show that the DDPG algorithm improves the click-through rate by 2.586% compared with the conventional recommendation algorithm. It shows that the algorithm designed in this paper has more obvious advantages in accurate recommendation by users. This paper effectively improves the efficiency of news dissemination by optimizing the push mode of intelligent news dissemination. In addition, the paper also deeply studies the innovative application of intelligent edge technology in news communication, which brings new ideas and practices to promote the development of news communication methods. Optimizing the push mode of intelligent news dissemination not only improves the user experience, but also provides strong support for the application of intelligent edge technology in this field, which has important practical application prospects.

Real-time AGV scheduling optimisation method with deep reinforcement learning for energy-efficiency in the container terminal yard

The increasing vessel size and automation level have shifted the productivity bottleneck of automated container terminals from the terminal side to the yard side. Operating an automated container terminal (ACT) yard with a big number of automated guided vehicles (AGV) is challenging due to the complexity and dynamics of the system, severely affecting the operational efficiency and energy use efficiency. In this paper, a hybrid multi-AGV scheduling algorithm is proposed to minimise the energy consumption and the total makespan of AGVs in an ACT yard. This framework first models the AGV scheduling process as a Markov decision process (MDP). Furthermore, a novel scheduling algorithm called MDAS is proposed based on multi-agent deep deterministic policy gradient (MADDPG) to facilitate online real-time scheduling decision-making. Finally, simulation experiments show that the proposed method can effectively enhance the operational efficiency and energy use performance of AGVs in ACT yards of various scales by comparing with benchmarking methods.

Multi-Objective Advantage Actor-Critic Algorithm for Hybrid Disassembly Line Balancing with Multi-Skilled Workers

The scheduling of disassembly lines is of great importance to achieve optimized productivity. In this paper, we address the Hybrid Disassembly Line Balancing Problem that combines linear disassembly lines and U-shaped disassembly lines, considering multi-skilled workers, and targeting profit and carbon emissions. In contrast to common approaches in reinforcement learning that typically employ weighting strategies to solve multi-objective problems, our approach innovatively incorporates non-dominated ranking directly into the reward function. The exploration of Pareto frontier solutions or better solutions is moderated by comparing performance between solutions and dynamically adjusting rewards based on the occurrence of repeated solutions. The experimental results show that the multi-objective Advantage Actor-Critic algorithm based on Pareto optimization exhibits superior performance in terms of metrics superiority in the comparison of six experimental cases of different scales, with an excellent metrics comparison rate of 70%. In some of the experimental cases in this paper, the solutions produced by the multi-objective Advantage Actor-Critic algorithm show some advantages over other popular algorithms such as the Deep Deterministic Policy Gradient Algorithm, the Soft Actor-Critic Algorithm, and the Non-Dominated Sorting Genetic Algorithm II. This further corroborates the effectiveness of our proposed solution.

Water management scheme based on prioritized deep deterministic policy gradient for proton exchange membrane fuel cells

AbstractTo effectively tackle the intricate and dynamic challenges encountered in proton exchange membrane fuel cells (PEMFCs), this paper introduces a model-free reinforcement learning approach to address its water management issue. Recognizing the limitations of conventional reinforcement learning methods such as Q-learning in handling the continuous actions and nonlinearity inherent in PEMFCs water management, we propose a prioritized deep deterministic policy gradient (DDPG) method. This method, rooted in the Actor-Critic framework, leverages double neural networks and prioritized experience replay to enable adaptive water management and balance. Additionally, we establish a PEMFCs water management platform and implement the prioritized DDPG method using "Tianshou", a modularized Python library for deep reinforcement learning. Through experimentation, the effectiveness of our proposed method is verified. This study contributes to advancing the understanding and management of water dynamics in PEMFCs, offering a promising avenue for enhancing their performance and reliability.

Energy management strategy with mutation protection for fuel cell electric vehicles

For hydrogen fuel cell vehicles, energy management strategies (EMS) are vital for balancing fuel cell and battery power, limiting fuel cell power, maintaining state of charge (SOC) fluctuation range and mitigating degradation. Reinforcement learning-based EMS, especially using deep Q-network (DQN) and deep deterministic policy gradient (DDPG), have demonstrated potential for enhancing the fuel economy of hybrid electric vehicles (HEV) and fuel cell electric vehicles (FCEV). This research proposes mutation protection DQN (MPD)-based EMS to improve hydrogen fuel economy and reduce fuel cell degradation under driving cycles with plenty of mutations. By quantifying mutation, exploring its relationship with driving conditions and integrating a mutation protection module with DQN, MPD-based EMS achieves approximately 11% and 6% better fuel economy compared to the other two learning-based EMS. Additionally, it also reduces fuel cell degradation by approximately 21% and 13%.

Distributed Computation Offloading and Power Control for UAV-Enabled Internet of Medical Things

The advancement of the Internet of Medical Things (IoMT) has led to the emergence of various health and emotion care services, e.g., health monitoring. To cater to increasing computational requirements of IoMT services, Mobile Edge Computing (MEC) has emerged as an indispensable technology in smart health. Benefiting from the cost-effectiveness of deployment, unmanned aerial vehicles (UAVs) equipped with MEC servers in Non-Orthogonal Multiple Access (NOMA) have emerged as a promising solution for providing smart health services in proximity to medical devices (MDs). However, the escalating number of MDs and the limited availability of communication resources of UAVs give rise to a significant increase in transmission latency. Moreover, due to the limited communication range of UAVs, the geographically-distributed MDs lead to workload imbalance of UAVs, which deteriorates the service response delay. To this end, this paper proposes a UAV-enabled Distributed computation Offloading and Power control method with Multi-Agent, named DOPMA, for NOMA-based IoMT environment. Specifically, this paper introduces computation and transmission queue models to analyze the dynamic characteristics of task execution latency and energy consumption. Moreover, a credit assignment scheme-based reward function is designed considering both system-level rewards and rewards tailored to each MD, and an improved multi-agent deep deterministic policy gradient algorithm is developed to derive offloading and power control decisions independently. Extensive simulations demonstrate that the proposed method outperforms existing schemes, achieving \(7.1\% \) reduction in energy consumption and \(16\% \) decrease in average delay.

Multiple DC Modulation Coordination Strategies Based on Transient Energy Function and Deep Deterministic Policy Gradient Algorithm

Ultra-low-frequency oscillation (ULFO) is a new problem of frequency stability in asynchronous network back-end power networks. The negative damping provided by hydropower units is the direct cause of ultra-low-frequency oscillation, and DC modulation is an effective means of suppressing system frequency oscillation. Based on the transient energy function and the limitation of DC modulation, this paper analyzes the conditions of suppressing system oscillation by DC modulation from the perspective of energy. The weight of unit energy participation is defined, and the calculation formula of transient oscillation energy in a certain oscillation mode is derived. DC modulation sensitivity is then used to create a DC modulation sequence table based on the Prony identification approach. Ultimately, the DC involved in the modulation is identified using the DC modulation sequence table and the transient oscillation energy, and the Deep Deterministic Policy Gradient (DDPG) algorithm is used to optimize the chosen DC modulation parameters.

From Nash Q-learning to nash-MADDPG: Advancements in multiagent control for multiproduct flexible manufacturing systems

The emergence of flexible manufacturing systems (FMS) capable of processing multiple product types is a result of the growing demand for product customization and personalization. Such multiproduct systems are characterized by a higher level of uncertainty and variability when compared to traditional manufacturing systems. This paper proposes a Nash integrated multiagent deep deterministic policy gradient method (Nash-MADDPG) to control the mobile robots’ assignment in a multiproduct FMS to enable intelligent decision-making, interaction, and dynamic learning capabilities. A mathematical model of a multiproduct FMS from a previous study is described, and system dynamic property is characterized by permanent production loss (PPL). Then, by observing PPL of the system and market demand for each product type, the multi-agent control scheme is developed to assign mobile robots to load/unload various product types at various machines. First, a Nash game is developed among the mobile robots and to improve the cooperation, a collaboration cost is defined. This collaboration cost is then used in the reward function of the multiagent deep deterministic policy gradient (MADDPG) algorithm. Second, the actions are jointly defined based on the action values of MADDPG, and the mobile robots’ strategies in the Nash game, which update the environment to a new state. The performance of the proposed method is verified by comparing it with conventional Nash Q-learning, vanilla MADDPG, Q-learning based single agent reinforcement learning (SARL) and a first-come-first-serve (FCFS) based control. The results demonstrate that the multi-agent control scheme under the proposed Nash-MADDPG is effective in dealing with cooperative FMS environment that involves complicated dynamics and uncertainties.

Reinforcement-Learning-Based Virtual Inertia Controller for Frequency Support in Islanded Microgrids

As the world grapples with the energy crisis, integrating renewable energy sources into the power grid has become increasingly crucial. Microgrids have emerged as a vital solution to this challenge. However, the reliance on renewable energy sources in microgrids often leads to low inertia. Renewable energy sources interfaced with the network through interlinking converters lack the inertia of conventional synchronous generators, and hence, need to provide frequency support through virtual inertia techniques. This paper presents a new control algorithm that utilizes the reinforcement learning agents Twin Delayed Deep Deterministic Policy Gradient (TD3) and Deep Deterministic Policy Gradient (DDPG) to support the frequency in low-inertia microgrids. The RL agents are trained using the system-linearized model and then extended to the nonlinear model to reduce the computational burden. The proposed system consists of an AC–DC microgrid comprising a renewable energy source on the DC microgrid, along with constant and resistive loads. On the AC microgrid side, a synchronous generator is utilized to represent the low inertia of the grid, which is accompanied by dynamic and static loads. The model of the system is developed and verified using Matlab/Simulink and the reinforcement learning toolbox. The system performance with the proposed AI-based methods is compared to conventional low-pass and high-pass filter (LPF and HPF) controllers.

Large-scale deep reinforcement learning method for energy management of power supply units considering regulation mileage payment

To improve automatic generation control (AGC) performance and reduce the wastage of regulation resources in interconnected grids including high-proportion renewable energy, a multi-area integrated AGC (MAI-AGC) framework is proposed to solve the coordination problem of secondary frequency regulation between different areas. In addition, a cocktail exploration multi-agent deep deterministic policy gradient (CE-MADDPG) algorithm is proposed as the framework algorithm. In this algorithm, the controller and power distributor of an area are combined into a single agent which can directly output the power generation command of different units. Moreover, the cocktail exploration strategy as well as various other techniques are introduced to improve the robustness of the framework. Through centralized training and decentralized execution, the proposed method can nonlinearly and adaptively derive the optimal coordinated control strategies for multiple agents and is verified on the two-area LFC model of southwest China and the four-area LFC model of the China Southern Grid (CSG).

Artificial intelligence-based position control: reinforcement learning approach in spring mass damper systems

This work examines the use of deep Reinforcement Learning (RL) in mass-spring system position control, providing a fresh viewpoint that goes beyond conventional control techniques. Mass-spring systems are widely used in many sectors and are basic models in control theory. The novel aspect of this approach is the thorough examination of the impact of several optimizer algorithms on the RL methodology, which reveals the optimal control tactics. The research applies a Deep Deterministic Policy Gradient (DDPG) algorithm for continuous action spaces, where the actor and critic networks are important components in assessing the agent’s performance. The RL agent is trained to follow a reference trajectory using the Simulink environment for system modeling. The study provides insights into the agent’s learning approach and performance optimization by evaluating the training process using force-time graphs, reward graphs, and Episode Manager charts. Furthermore, the effect of different combinations of optimizers on the control performance of the agent is examined. The outcomes highlight the importance of optimizer selection in the learning process by revealing significant variations in training times. As a result, a better understanding of the relationship between various optimizers and control performance is provided by this study’s novel application of reinforcement learning in mass-spring system control. The results raise the possibility of more potent methods for controlling complex systems and add to the expanding field of study at the interface of control theory and deep learning.

Integrating Multi-Agent Deep Deterministic Policy Gradient and Go-Explore for Enhanced Reward Optimization

The field of Multi-Agent Reinforcement Learning (MARL) continues to advance with the development of new and effective methods. This research is centered on two prominent approaches within this field: Multi-Agent Deep Deterministic Policy Gradient (MADDPG) and Go-Explore. The study explores the synergistic potential of combining these two methodologies to enhance rewards for individual agents as well as for agent groups. In the course of this research, MADDPG is introduced into the experimental environment, providing agents with both actor networks (policy networks) and critic networks (Q networks) to implement the actor-critic model. Additionally, each individual agent is equipped with a Go-Explore network, empowering them to conduct deeper explorations of the environment and accumulate rewards at an accelerated rate, often resulting in higher overall rewards. This novel approach emphasizes achieving a balance between individual and collaborative rewards, offering a promising avenue for optimizing multi-agent systems. The results of this study demonstrate that the combined method exhibits notable advantages in certain scenarios. Specifically, it showcases a higher rate of reward accumulation and improved overall performance. This research contributes to the MARL domain by highlighting the potential of combining MADDPG and Go-Explore to enhance the efficiency and effectiveness of multi-agent systems.

Deep Deterministic Policy Gradient (DDPG) Agent-Based Sliding Mode Control for Quadrotor Attitudes

A novel reinforcement deep learning deterministic policy gradient agent-based sliding mode control (DDPG-SMC) approach is proposed to suppress the chattering phenomenon in attitude control for quadrotors, in the presence of external disturbances. First, the attitude dynamics model of the quadrotor under study is derived, and the attitude control problem is described using formulas. Second, a sliding mode controller, including its sliding mode surface and reaching law, is chosen for the nonlinear dynamic system. The stability of the designed SMC system is validated through the Lyapunov stability theorem. Third, a reinforcement learning (RL) agent based on deep deterministic policy gradient (DDPG) is trained to adaptively adjust the switching control gain. During the training process, the input signals for the agent are the actual and desired attitude angles, while the output action is the time-varying control gain. Finally, the trained agent mentioned above is utilized in the SMC as a parameter regulator to facilitate the adaptive adjustment of the switching control gain associated with the reaching law. The simulation results validate the robustness and effectiveness of the proposed DDPG-SMC method.

DBlock-RLB: An energy efficient framework for intelligent routing and trading based load balancing in SDWSN environment

Wireless Sensor Networks (WSNs) is an emerging field that enables them to work in complex scenarios with enough flexibility. The WSN systems are improved in terms of network flexibility by integrating Software Defined Network (SDN) to obtain SDWSN environment. However, security and energy efficiency are the major issues in the SDWSN environment which leads to low network lifetime. To resolve the existing issues, we proposed DAG-based BLOCKchain-Routing and Load Balancing Model (DBLOCK-RLB). The proposed approach performs routing and load balancing in the SDWSN environment in an energy efficient manner with high security. The SDWSN environment is composed of four major planes as data plane, switch plane, control plane, and application plane. The sensor nodes in the data plane are authenticated priory to ensure legitimacy and reduce unwanted network traffics by adopting Camellia Encryption Algorithm (CAE) which provides secret key to the user by considering several metrics. The authenticated sensor nodes are clustered to reduce energy consumption using Adaptive Threshold based Network Partitioning in which the circular SDWSN environment is split into equal parts and optimal Cluster Head selection is done by considering clustering metrics. From the clustered network, the CH is responsible for intelligent routing. For intelligent routing, we utilize Dual Agent-Twin Delayed Deep Deterministic Policy Gradient (DA-TD3) in which one agent for optimal forwarder selection and another agent for optimal route selection. After routing, the load balancing is done to reduce the network overhead and scalability issues. The proposed work performs trading based load balancing by utilizing Stackelberg Game Model in which all the local controllers are considered as followers and global controller is considered as leaders. All the transactions are stored in the DAG blockchain to reduce the scalability and Proof-of Authentication (PoAH) used to verify the transactions. The proposed work is simulated using NS-3.26 simulation tool and compared with several existing works by validation metrics like throughput at 2.55(mbps), latency at 27.1(ms), packet delivery ratio at 91.5(%), network Lifetime at 546.5(s), and energy consumption at 14.5 (J). The results show that the proposed work outperforms better than the existing works.

Autonomous port management based AGV path planning and optimization via an ensemble reinforcement learning framework

The rapid development of shipping trade pushes automated container terminals toward the direction of intelligence, safety and efficiency. In particular, the formulation of AGV scheduling tasks and the safety and stability of transportation path is an important part of port operation and management, and it is one of the basic tasks to build an intelligent port. Existing research mainly focuses on collaborative operation between port equipment and path optimization under environmental perception, while there is relatively little research on optimization of path smoothness and safety. Therefore, we propose a path optimization model based on the artificial potential field and twin delayed deep deterministic policy gradient (APF-TD3) framework for the port environment. Firstly, we obtain the scheduling task plan of a single AGV by enumeration. Secondly, according to the artificial potential field (APF) algorithm to generate repulsion for obstacles in the harbor and attraction for container storage at the target point with the position information of the AGV as the input data of the reinforcement learning algorithm is inputted into the twin delayed deep deterministic policy gradient algorithm (TD3). Then TD3 selects the optimal action strategy for the AGV according to the input AGV state information and the designed reward mechanism as well as executes the action. Through repeated execution, the optimal action for the next step is selected at each point to generate a path with start and end points. We validate the model by simulating the scale of containerized cargo in the port i.e. small scale, medium scale and large scale scenes. The experimental results show that the method has the shortest path length of 27.519 m, 270.847 m, and 496.389 m compared to artificial potential field and deep deterministic policy gradient (APF-DDPG), APF, and rapidly-exploring random tree (RRT) algorithms, which also have significant advantages in terms of path security and path smoothness. This framework could respond to the scheduling and transportation tasks of single AGV in different environmental layouts and guarantee the smoothness and safety of the path based on the optimization of the path, which promotes the efficient operation and management of ports.

An optimal solutions-guided deep reinforcement learning approach for online energy storage control

As renewable energy becomes more prevalent in the power grid, energy storage systems (ESSs) are playing an ever-increasingly crucial role in mitigating short-term supply–demand imbalances. However, the operation and control of ESS are not straightforward, given the ever-changing electricity prices in the market environment and the stochastic and intermittent nature of renewable energy generations, which respond to real-time load variations. In this paper, we propose a deep reinforcement learning (DRL) approach to address the electricity arbitrage problem associated with optimal ESS management. First, we analyze the structure of the optimal offline ESS control problem using the mixed-integer linear programming (MILP) formulation. This formulation identifies optimal control actions to absorb excess renewable energy and perform price arbitrage strategies. To tackle the uncertainties inherent in the prediction data, we then recast the online ESS control problem into a Markov Decision Process (MDP) framework and develop the DRL approach, which involves integrating the optimal offline control solution obtained from the training data into the training process and introducing noise to the state transitions. Unlike typical offline DRL training over a long time interval, we employ the Deep Deterministic Policy Gradient (DDPG) and Proximal Policy Optimization (PPO) algorithms with smaller neural networks training over a short time interval. Numerical studies demonstrate the promising potential of the proposed DRL-enabled approach for achieving better online control performance than the model predictive control (MPC) method under different price errors. This highlights the sample efficiency and robustness of our DRL approaches in managing ESS for electricity arbitrage.