Hierarchical Deep Reinforcement Learning Research Articles

An improved hierarchical deep reinforcement learning algorithm for multi-intelligent vehicle lane change

Lane change is a pivotal technology in autonomous driving, playing a significant role in improving traffic conditions. The control task of lane change becomes exceptionally complex for coordination of multiple intelligent vehicles, especially in unpredictable situations. To address this issue, the paper proposes a novel multi-intelligent vehicle lane change method (MVLC) based on Deep Reinforcement Learning (DRL). The MVLC features a hierarchical framework that includes a lower-layer lane change controller and an upper-layer reinforcement learning decision-making method. In the lower layer, a lane change controller is designed to execute motion control for a single vehicle, consisting of a lateral controller utilizing dueling double deep Q-network to make lane change decision and a longitudinal controller that employs Intelligent Driver Model (IDM) to follow the preceding vehicle. In the upper layer, a model-free DRL method is used for simultaneous control of multiple intelligent vehicles. To learn the optimal policy, a comprehensive reward function is designed. Finally, experiments are conducted in mixed traffic to valid the effectiveness of the proposed method. Results show that MVLC achieves secure and timely lane changes for multiple vehicles. Therefore, the method is expected to have significant potential for improving overall traffic efficiency and mitigating traffic congestion.

Multiagent Hierarchical Deep Reinforcement Learning for Operation Optimization of Grid-Interactive Efficient Commercial Buildings

Multiagent Hierarchical Deep Reinforcement Learning for Operation Optimization of Grid-Interactive Efficient Commercial Buildings

The AI circular hydrogen economist: Hydrogen supply chain design via hierarchical deep multi-agent reinforcement learning

Hydrogen supply chain (HSC) consists of various production, conditioning, transportation, storage, and distribution processes, all of which require extensive computational resources for precise modelling. In this context, artificial intelligence (AI) is emerging as a pivotal tool for addressing model-based decision-making challenges, courtesy of its rapid and efficient computational capabilities. This paper proposes a comprehensive HSC model consisting of an economic policy planner, a wholesale hydrogen market, a power supply system, a hydrogen distribution system (HDS), and hydrogen refueling stations (HRSs). It leverages an AI circular hydrogen economist approach based on a hierarchical deep multi-agent reinforcement learning (MARL) algorithm, offering a new alternative to traditional multi-objective bi-level optimization platforms. The model incorporates green, blue, and gray hydrogen production processes as viable hydrogen production pathways, with the hierarchical MARL’s agents representing the HDS and HRSs at two decision-making levels. Energy requirements are met through a combination of on-site renewable energy sources, the main power grid, or distributed power generation systems. Several HSC scenarios are examined with respect to various combinations of green hydrogen supply rates and carbon credits granted to them under optimum conditions. The results showed that the developed hierarchical MARL has the potential to replace mathematical programming (MP), uncovering a new economic-environmental trade-off between profit of HDS and operational costs of HRSs. Notably, green hydrogen transactions exponentially increase within the supply chain as the carbon credits exceed $1 per kilogram of hydrogen. While this research focuses on optimizing daily operations within the HSC, future efforts can aim to extend this optimization to forecasted annual operations.

Learning to route and schedule links in reconfigurable networks

This paper considers networks with a reconfigurable topology with so called 60 GHz dynamic links that can be activated or disabled over time. A fundamental problem is to jointly determine which 60 GHz dynamic links are active and the route chosen by source nodes over time. To this end, this paper outlines a hierarchical deep reinforcement learning solution that can be used to compute the optimal policy that determines for each time slot (i) active dynamic links, and (ii) the route used by each source–destination pair. The results show that the proposed approach results in a maximum average queue length that is 80% shorter than non-learning methods.

Multi-objective explainable smart dispatch for integrated energy system based on an explainable MO-RL method

With the rapid development of renewable energy penetration and the transition to high-efficiency, low-carbon-footprint energy systems, renewable-based heat-electricity integrated energy systems (RHE-IES) have become a viable method. The dispatch and operation scheduling of the RHE-IES is more complex than before because a well-operated RHE-IES should be economically favorable, environmentally friendly, and feasible with respect to security constraints. However, it is not easy to balance these trade-offs with uncertainties. In this work, a multi-objective hierarchical deep reinforcement learning (H-DRL) method is proposed to address the dispatch of RHE-IES, and it is also explainable due to the design of the algorithmic structure, the decoupling of rewards, and the analysis of feature importance. First, the dispatch and operation scheduling problem of the RHE-IES is formulated considering economic, environmental, and security constraints and is converted to a Markov decision process. Second, a novel H-DRL method is proposed with a multi-critic, single-actor structure, where each objective is handled by a designated deep critic network with decoupled rewards, and the action value functions of different objectives are hierarchically optimized by the actor network. H-DRL can be trained in a comprehensive environment offline and provide fast online decisions considering the uncertainties of renewable generation. Third, a shapley additive explanations method is carried out utilizing the explainable nature of the H-DRL structure and the loss function design. The simulation results and case studies show the effectiveness of the proposed method in terms of balancing the tradeoffs of different objectives, overcoming uncertainties and enhancing AI interpretability.

Hierarchical deep reinforcement learning for self-adaptive economic dispatch

It is challenging to accurately model the overall uncertainty of the power system when it is connected to large-scale intermittent generation sources such as wind and photovoltaic generation due to the inherent volatility, uncertainty, and indivisibility of renewable energy. Deep reinforcement learning (DRL) algorithms are introduced as a solution to avoid modeling the complex uncertainties and to adapt the fluctuation of uncertainty by interacting with the environment and using feedback to continuously improve their strategies. However, the large-scale nature and uncertainty of the system lead to the sparse reward problem and high-dimensional space issue in DRL. A hierarchical deep reinforcement learning (HDRL) scheme is designed to decompose the process of solving this problem into two stages, using the reinforcement learning (RL) agent in the global stage and the heuristic algorithm in the local stage to find optimal dispatching decisions for power systems under uncertainty. Simulation studies have shown that the proposed HDRL scheme is efficient in solving power system economic dispatch problems under both deterministic and uncertain scenarios thanks to its adaptation system uncertainty, and coping with the volatility of uncertain factors while significantly improving the speed of online decision-making.

Privacy-preserving multi-level co-regulation of VPPs via hierarchical safe deep reinforcement learning

Large amounts of distributed energy sources have brought challenges in terms of the safe and stable operation of the power grid. As a key technology between user-side energy resources and the distribution network (DN), how to realize the online coordinated scheduling of VPP and DN as well as the real-time response strategy of distributed equipment (DE) within VPP is the focus of this study. Thus, the hierarchical deep reinforcement learning (DRL) Hierarchical-TD3 algorithm is designed based on the unified modeling of adjustable space to achieve the real-time economic scheduling of VPPs. The upper layer DN considers the network security constraints and solves the economic scheduling model of VPPs based on the single-agent TD3 algorithm. Based on the scheduling instructions from the upper layer, the lower layer VPPs consider the requirements of privacy protection and control autonomy and realize real-time response of the DE within VPP via multi-agent MATD3 algorithm. Numerical results in the modified 33-nodes system show that the proposed Hierarchical-TD3 algorithm can achieve privacy protection and the coordinated scheduling of VPP and DE to reduce the operating cost. It differs from the optimal value by only 1.46% but can achieve online decision-making on the millisecond scale. Compared with the traditional centralized and decentralized DRL algorithms, the total cost is reduced by 10.15% and 5.52% respectively. Compared with the traditional soft-constraint method, there is no constraint violation during the training and testing phases. Finally, the actual 116-nodes testing system validates the scalability of the proposed Hierarchical-TD3 algorithm.

Deep-Reinforcement-Learning-Based Collision Avoidance of Autonomous Driving System for Vulnerable Road User Safety

The application of autonomous driving system (ADS) technology can significantly reduce potential accidents involving vulnerable road users (VRUs) due to driver error. This paper proposes a novel hierarchical deep reinforcement learning (DRL) framework for high-performance collision avoidance, which enables the automated driving agent to perform collision avoidance maneuvers while maintaining appropriate speeds and acceptable social distancing. The novelty of the DRL method proposed here is its ability to accommodate dynamic obstacle avoidance, which is necessary as pedestrians are moving dynamically in their interactions with nearby ADSs. This is an improvement over existing DRL frameworks that have only been developed and demonstrated for stationary obstacle avoidance problems. The hybrid A* path searching algorithm is first applied to calculate a pre-defined path marked by waypoints, and a low-level path-following controller is used under cases where no VRUs are detected. Upon detection of any VRUs, however, a high-level DRL collision avoidance controller is activated to prompt the vehicle to either decelerate or change its trajectory to prevent potential collisions. The CARLA simulator is used to train the proposed DRL collision avoidance controller, and virtual raw sensor data are utilized to enhance the realism of the simulations. The model-in-the-loop (MIL) methodology is utilized to assess the efficacy of the proposed DRL ADS routine. In comparison to the traditional DRL end-to-end approach, which combines high-level decision making with low-level control, the proposed hierarchical DRL agents demonstrate superior performance.

Progressive Hierarchical Deep Reinforcement Learning for defect wafer test

Wafer testing plays a critical role in the wafer fabrication process. It involves using a probe to inspect all the wafers on a map, which presents a Coverage Path Planning (CPP) problem. Deep Reinforcement Learning (DRL) has emerged as a state-of-the-art (SOTA) solution for automating regular wafer testing. However, in the presence of production defects, the wafer maps become irregular, requiring more advanced probe strategies. Unfortunately, SOTA DRL methods cannot be directly applied to defect wafer testing due to high computational complexity, low sample efficiency, and limited transferability between tasks. To address these challenges, we propose a novel DRL solution called the Progressive Hierarchical DRL method for automated defect wafer testing. Specifically, our method utilizes a Visual Memory Attention Model (VMAM) to create a high-level strategy, referred to as a manager, which generates a goal specifying the most crucial image region for the task. Subsequently, a low-level strategy, referred to as a worker, generates precise cruising actions to navigate to the designated area. Additionally, we employ transfer learning to generalize the method to handle more complex defect wafer maps. Our approach outperforms seven SOTA DRL baselines in the domains of CPP and wafer testing. It achieves significant improvements in task duration and wafer visit overlap, ranging from 6.0% to 30.5%. Furthermore, we explore the application of our method to a dual-probe defect wafer testing system.

Interactive learning for multi-finger dexterous hand: A model-free hierarchical deep reinforcement learning approach

When a multi-fingered dexterous hand interacts with the external environment, it encounters various challenges, including the utilization of complex control techniques and the intricate coordination of finger motion sequences. Previous studies have primarily concentrated on investigating the interaction between multi-fingered dexterous hands and external objects, usually using model-based control or model-free reinforcement learning techniques. However, during practical implementation, various constraining factors are encountered, such as intricate modeling and limited interaction capabilities. In practical scenarios, the utilization of multi-fingered dexterous hands is imperative for the swift and efficient execution of a wide range of interactive tasks, including but not limited to throwing a ball and playing rock-paper-scissors. These tasks require skilled manual dexterity to demonstrate both precise control and quick responsiveness. To tackle this issue, we propose a hierarchical control approach for multi-fingered dexterous hands with interactive functionalities, utilizing model-free deep reinforcement learning. The complex interaction task is decomposed into simple sub-tasks using hierarchical strategy and action primitive decomposition, which effectively reduces the complexity of the action space, and achieves the motion planning and end finger trajectory control of dexterous hand. In a simulated environment, the aforementioned method has successfully executed interactive tasks, including ball throwing and playing rock-paper-scissors. It achieved a maximum normalized reward of 0.83 and an 84 % success rate. These results are noteworthy in terms of both control accuracy and response speed. This study offers novel insights into the effective resolution of the intricate challenges associated with interactions involving multi-fingered dexterous hands and human-computer interaction.

Distributed Hierarchical Deep Reinforcement Learning for Large-Scale Grid Emergency Control

Distributed Hierarchical Deep Reinforcement Learning for Large-Scale Grid Emergency Control

Utilizing Deep Reinforcement Learning, EGO-Swarm Parameter Control determines the path of the drone

The recent attention garnered by the EGO-Swarm system is due to its successful generation of smooth drone flight paths in complex real-world environments. Among the various parameter values utilized by EGO-Swarm for path generation, the maximum speed and maximum acceleration of the drone play a critical role in enhancing flight performance. However, in the current EGO-Swarm setup, these parameter values remain fixed during path creation, despite the potential need for real-time adjustments in dynamic environments. Therefore, there is a need to dynamically adapt these values to the evolving real-time environment. This paper proposes a novel algorithm that dynamically adjusts the maximum speed and maximum acceleration using hierarchical deep reinforcement learning in response to real-time environmental changes. To assess the effectiveness of the proposed method, a comparative experiment is conducted between the existing EGO-Swarm algorithm and the proposed algorithm in a ROS simulation. The experimental results reveal that the proposed algorithm surpasses the existing EGO-Swarm algorithm in terms of average speed improvement and path length.(Zhou et al., 2021)

The SoS conductor: Orchestrating resources with iterative agent‐based reinforcement learning

AbstractWe introduce a novel resource management approach for Systems of Systems (SoS), utilizing hierarchical deep reinforcement learning, iterating with agent‐based simulation. A key innovation of this method is its ability to balance top‐down SoS management with the autonomy of individual systems. This is achieved by dynamically allocating resources to each system, thereby modifying the range of options they can autonomously choose from. This dynamic option adjustment is a powerful approach to managing the trade‐off between centralized efficiency and decentralized autonomous actions of the systems, enabling the SoS to maintain the systems' autonomy while ensuring efficient SoS governance. The method, validated through a case study, not only demonstrates the potential and efficacy of the learning framework but also reveals how, using this method, minor performance sacrifices can lead to substantial improvements in resource efficiency.

An adaptive multi-objective multi-task scheduling method by hierarchical deep reinforcement learning

Actual manufacturing process scheduling in enterprise alliances are multi-task scheduling problems involving dynamic factors, and the competition and conflict for manufacturing resources also exist between multi-tasks. How to perform adaptive multi-objective scheduling of multi-tasks based on the real-time state of the manufacturing environment becomes critical. Therefore, this paper constructs an adaptive multi-task multi-objective scheduling considering resource competition and conflict among tasks (AMMS-RCCT) model based on the enterprise alliance value net, and adopts a hybrid strategy of “parallel+serial” to resolve conflicts while reducing the waiting time of tasks. With the objective of optimizing the total manufacturing time and total manufacturing cost, an adaptive multi-objective deep Q network (AMDQN) is proposed to solve the AMMS-RCCT model. AMDQN is based on a two-hierarchy deep reinforcement learning architecture containing a front controller deep Q network (C-DQN) and a back actuator deep Q network (A-DQN), which performs hierarchical decision-making on optimization objectives and scheduling rules to achieve compromise between multiple objectives while reducing the complexity for optimal selection scheduling rules. For the two optimization objectives of time and cost, two reward algorithms are proposed by introducing two metrics, the estimated tardiness rate and the estimated overspend rate, which guide the A-DQN to learn and adjust the scheduling rules according to the state changes. Besides, nine composite scheduling rules are designed to adapt to the dynamic manufacturing environment from multiple dimensions such as task urgency and completion rate as well as manufacturing resource utilization and cost. Finally, AMDQN is experimentally compared with the proposed nine composite scheduling rules, scheduling rules in existing research, and other scheduling methods based on reinforcement learning in simulated manufacturing environments with different numbers of tasks, subtasks, and manufacturing cells. The experimental results verify the effectiveness and superiority of AMDQN for multi-objective adaptive scheduling in multi-task scheduling problems.

Electric vehicle charging navigation strategy in coupled smart grid and transportation network: A hierarchical reinforcement learning approach

Most of the existing electric vehicle (EV) charging navigation methods do not simultaneously take into account the electric vehicle charging destination optimization and path planning. Moreover, they are unable to provide online real-time decision-making under a variety of uncertain factors. To address these problems, this paper first establishes a bilevel stochastic optimization model for EV charging navigation considering various uncertainties, and then proposes an EV charging navigation method based on the hierarchical enhanced deep Q network (HEDQN) to solve the above stochastic optimization model in real-time. The proposed HEDQN contains two enhanced deep Q networks, which are utilized to optimize the charging destination and charging route path of EVs, respectively. Finally, the proposed method is simulated and validated in two urban transportation networks. The simulation results demonstrate that compared with the Dijkstra shortest path algorithm, single-layer deep reinforcement learning algorithm, and traditional hierarchical deep reinforcement learning algorithm, the proposed HEDQN algorithm can effectively reduce the total charging cost of electric vehicles and realize online real-time charging navigation of electric vehicles, that shows excellent generalization ability and scalability.

Energy-efficient access point clustering and power allocation in cell-free massive MIMO networks: a hierarchical deep reinforcement learning approach

Cell-free massive multiple-input multiple-output (CF-mMIMO) has attracted considerable attention due to its potential for delivering high data rates and energy efficiency (EE). In this paper, we investigate the resource allocation of downlink in CF-mMIMO systems. A hierarchical depth deterministic strategy gradient (H-DDPG) framework is proposed to jointly optimize the access point (AP) clustering and power allocation. The framework uses two-layer control networks operating on different timescales to enhance EE of downlinks in CF-mMIMO systems by cooperatively optimizing AP clustering and power allocation. In this framework, the high-level processing of system-level problems, namely AP clustering, enhances the wireless network configuration by utilizing DDPG on the large timescale while meeting the minimum spectral efficiency (SE) constraints for each user. The low layer solves the link-level sub-problem, that is, power allocation, and reduces interference between APs and improves transmission performance by utilizing DDPG on a small timescale while meeting the maximum transmit power constraint of each AP. Two corresponding DDPG agents are trained separately, allowing them to learn from the environment and gradually improve their policies to maximize the system EE. Numerical results validate the effectiveness of the proposed algorithm in term of its convergence speed, SE, and EE.

On Designing Multi-UAV aided Wireless Powered Dynamic Communication via Hierarchical Deep Reinforcement Learning

On Designing Multi-UAV aided Wireless Powered Dynamic Communication via Hierarchical Deep Reinforcement Learning

Spatial memory-augmented visual navigation based on hierarchical deep reinforcement learning in unknown environments

Visual navigation in unknown environments poses significant challenges due to the presence of many obstacles and low-texture scenes. These factors may cause frequent collisions and tracking failure of feature-based visual Simultaneous Localization and Mapping (vSLAM). To avoid these issues, this paper proposes a spatial memory-augmented visual navigation system that combines a vSLAM module, a conventional global planner module, and a Hierarchical Reinforcement Learning (HRL)-based local planner module. Firstly, a real-time vSLAM named Salient-SLAM is proposed to augment the performance of visual navigation. Salient-SLAM creates a navigation mapping thread by combining a saliency prediction model to build a navigation map that categorizes environmental regions as occupied, explored, or noticeable. Spatial memory that contains spatial abstraction and saliency information of the environment can be further formed by encoding navigation maps, which helps the agent determine an optimal path towards its destination. An open-sourced saliency dataset is proposed to train the saliency prediction model by mimicking the visual attention mechanism. Secondly, a HRL method is proposed to automatically decompose local planning into a high-level policy selector and several low-level policies, where the latter produces actions to interact with the environment. We maximize entropy and minimize option correlation in learning low-level policies, aiming at acquiring diverse and independent behaviors. The simulation results show that the proposed HRL method outperforms competitive baselines by 6.29–10.85 % on Success Rate (SR) and 3.87–11.1 % on Success weighted by Path Length (SPL) metrics. By incorporating the spatial memory, SR, and SPL metrics can be augmented by an average of 9.85 % and 10.89 %, respectively.

Hierarchical Reinforcement Learning for Air Combat at DARPA's AlphaDogfight Trials

Autonomous control in high-dimensional, continuous state spaces is a persistent and important challenge in the fields of robotics and artificial intelligence. Because of high risk and complexity, the adoption of AI for autonomous combat systems has been a long-standing difficulty. In order to address these issues, DARPA's AlphaDogfight Trials (ADT) program sought to vet the feasibility of and increase trust in AI for autonomously piloting an F-16 in simulated air-to-air combat. Our submission to ADT solves the high-dimensional, continuous control problem using a novel hierarchical deep reinforcement learning approach consisting of a high-level policy selector and a set of separately trained low-level policies specialized for excelling in specific regions of the state space. Both levels of the hierarchy are trained using off-policy, maximum entropy methods with expert knowledge integrated through reward shaping. Our approach outperformed human expert pilots and achieved a second-place rank in the ADT championship event. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Impact Statement–</i> Significant performance milestones in reinforcement learning have been achieved in recent years, with autonomous agents demonstrating super-human performance across a wide variety of tasks. Before these algorithms can be extensively deployed in real-world defense applications, a greater level of trust must first be achieved. ADT was an important step towards developing the trust necessary to operationalize these algorithms, by demonstrating their effectiveness on a foundational yet relevant problem in a high-fidelity simulation environment. Developed for the program, our hierarchical reinforcement learning agent was designed alongside of and competed against active fighter pilots, and ultimately defeated a graduate of the United States Air Force's F-16 Weapons Instructor Course in match play.

Toward Scalable and Efficient Hierarchical Deep Reinforcement Learning for 5G RAN Slicing

Toward Scalable and Efficient Hierarchical Deep Reinforcement Learning for 5G RAN Slicing