Hierarchical Reinforcement Learning Algorithm Research Articles

Electric vehicle charging navigation strategy in coupled smart grid and transportation network: A hierarchical reinforcement learning approach

Most of the existing electric vehicle (EV) charging navigation methods do not simultaneously take into account the electric vehicle charging destination optimization and path planning. Moreover, they are unable to provide online real-time decision-making under a variety of uncertain factors. To address these problems, this paper first establishes a bilevel stochastic optimization model for EV charging navigation considering various uncertainties, and then proposes an EV charging navigation method based on the hierarchical enhanced deep Q network (HEDQN) to solve the above stochastic optimization model in real-time. The proposed HEDQN contains two enhanced deep Q networks, which are utilized to optimize the charging destination and charging route path of EVs, respectively. Finally, the proposed method is simulated and validated in two urban transportation networks. The simulation results demonstrate that compared with the Dijkstra shortest path algorithm, single-layer deep reinforcement learning algorithm, and traditional hierarchical deep reinforcement learning algorithm, the proposed HEDQN algorithm can effectively reduce the total charging cost of electric vehicles and realize online real-time charging navigation of electric vehicles, that shows excellent generalization ability and scalability.

Hierarchical Reinforcement Learning for UAV-PE Game With Alternative Delay Update Method.

This article proposes a novel hierarchical reinforcement learning (HRL) algorithm for unmanned aerial vehicle pursuit-evasion (UAV-PE) game systems with an alternative delay update (ADU) method. In the proposed algorithm, the approximate solutions of the UAV-PE game problem are derived from a hierarchical learning process, which relies on a zero-sum game process of kinematics and a corresponding optimal process of dynamics. In this case, deep neural networks (NNs) are used to approximate the policy and value functions of UAV-PE game systems in kinematics and dynamics level. Furthermore, the ADU method is adopted to improve the training efficiency of deep NN by fixing one player of the UAV-PE game systems to form a stable environment. The goal of this article is to develop an HRL algorithm with an ADU method for obtaining approximate Nash equilibrium (NE) solutions of the considered UAV-PE game systems which are subjected to the coupling of kinematics and dynamics. Subsequently, sufficient conditions are provided for analyzing the convergence and optimality of the proposed HRL algorithm. Moreover, the inequalities of overload are obtained to guarantee that the state of dynamics tracks with the control input of kinematics in UAV-PE game systems. Finally, simulation examples are provided to demonstrate the feasibility and usefulness of the proposed HRL algorithm and ADU method.

Goal-Conditioned Hierarchical Reinforcement Learning With High-Level Model Approximation.

Hierarchical reinforcement learning (HRL) exhibits remarkable potential in addressing large-scale and long-horizon complex tasks. However, a fundamental challenge, which arises from the inherently entangled nature of hierarchical policies, has not been understood well, consequently compromising the training stability and exploration efficiency of HRL. In this article, we propose a novel HRL algorithm, high-level model approximation (HLMA), presenting both theoretical foundations and practical implementations. In HLMA, a Planner constructs an innovative high-level dynamic model to predict the k -step transition of the Controller in a subtask. This allows for the estimation of the evolving performance of the Controller. At low level, we leverage the initial state of each subtask, transforming absolute states into relative deviations by a designed operator as Controller input. This approach facilitates the reuse of subtask domain knowledge, enhancing data efficiency. With this designed structure, we establish the local convergence of each component within HLMA and subsequently derive regret bounds to ensure global convergence. Abundant experiments conducted on complex locomotion and navigation tasks demonstrate that HLMA surpasses other state-of-the-art single-level RL and HRL algorithms in terms of sample efficiency and asymptotic performance. In addition, thorough ablation studies validate the effectiveness of each component of HLMA.

Price-Matching-Based Regional Energy Market With Hierarchical Reinforcement Learning Algorithm

Price-Matching-Based Regional Energy Market With Hierarchical Reinforcement Learning Algorithm

DisTop: Discovering a Topological Representation to Learn Diverse and Rewarding Skills

An efficient way for a deep reinforcement learning (DRL) agent to explore can be to learn a set of skills that achieves a uniform distribution of states. Following this, we introduce DisTop, a new model that simultaneously learns diverse skills and focuses on improving rewarding skills. DisTop progressively builds a discrete topology of the environment using an unsupervised contrastive loss, a growing network and a goal-conditioned policy. Using this topology, a state-independent hierarchical policy can select where the agent has to keep discovering skills in the state space and explicitly forget skills unrelated to tasks. In turn, the new set of visited states allows an improved learnt representation and the learning loop continues. Our experiments emphasize that DisTop is agnostic to the ground state representation and that the agent can discover the topology of its environment whether the states are high-dimensional binary data, images, or proprioceptive inputs. We demonstrate that this paradigm is competitive on MuJoCo benchmarks with state-of-the-art algorithms on both single-task dense rewards and diverse skill discovery. By combining these two aspects, we show that DisTop outperforms a state-of-the-art hierarchical reinforcement learning (HRL) algorithm when rewards are sparse. We believe DisTop opens new perspectives by showing that bottom-up skill discovery combined with representation learning can tackle different hard reward settings.

A Hierarchical Reinforcement Learning Algorithm Based on Attention Mechanism for UAV Autonomous Navigation

A Hierarchical Reinforcement Learning Algorithm Based on Attention Mechanism for UAV Autonomous Navigation

Towards efficient long-horizon decision-making using automated structure search method of hierarchical reinforcement learning for edge artificial intelligence

Hierarchical reinforcement learning (HRL) is a promising approach for efficiently solving various long-horizon decision-making tasks in the Internet of Things (IoT) domain. However, HRL algorithms are known to rely on expert knowledge to preset an appropriate hierarchical structure for different IoT tasks, which leads to higher trial costs and limits its wider application. In this paper, we propose a new method called DHRL (Dynamic-Level Hierarchical Reinforcement Learning) and it is able to adaptively search for the optimal hierarchical structure while maintaining the generality of framework design. DHRL incorporates an embedded exploration and exploitation mechanism that effectively solves the challenges caused by dependence between different levels and achieves a balance between maximizing benefits and current evaluation accuracy. Nonetheless, the more exploration processes inevitably has a negative impact on the performance. To mitigate this influences, we propose a synchronous training architecture to support DHRL operating in a distributed and parallel manner, in which the adaptive evolutionary method is also introduced to accelerate the convergence. Extensive experimental evaluations are conducted to demonstrate the effectiveness of our theory and method.

Deep hierarchical reinforcement learning based formation planning for multiple unmanned surface vehicles with experimental results

In this paper, a novel multi-USV formation path planning algorithm is proposed based on deep reinforcement learning. First, a goal-based hierarchical reinforcement learning algorithm is designed to improve training speed and resolve planning conflicts within the formation. Second, an improved artificial potential field algorithm is designed in the training process to obtain the optimal path planning and obstacle avoidance learning scheme for multi-USVs in the determined perceptual environment. Finally, a formation geometry model is established to describe the physical relationships among USVs, and a composite reward function is proposed to guide the training. Numerous simulation tests are conducted, and the effectiveness of the proposed algorithm are further validated through the NEU-MSV01 experimental platform with a combination of parameterized Line of Sight (LOS) guidance.

Hierarchical Reinforcement Learning for Crude Oil Supply Chain Scheduling

Crude oil resource scheduling is one of the critical issues upstream in the crude oil industry chain. It aims to reduce transportation and inventory costs and avoid alerts of inventory limit violations by formulating reasonable crude oil transportation and inventory strategies. Two main difficulties coexist in this problem: the large problem scale and uncertain supply and demand. Traditional operations research (OR) methods, which rely on forecasting supply and demand, face significant challenges when applied to the complicated and uncertain short-term operational process of the crude oil supply chain. To address these challenges, this paper presents a novel hierarchical optimization framework and proposes a well-designed hierarchical reinforcement learning (HRL) algorithm. Specifically, reinforcement learning (RL), as an upper-level agent, is used to select the operational operators combined by various sub-goals and solving orders, while the lower-level agent finds a viable solution and provides penalty feedback to the upper-level agent based on the chosen operator. Additionally, we deploy a simulator based on real-world data and execute comprehensive experiments. Regarding the alert number, maximum alert penalty, and overall transportation cost, our HRL method outperforms existing OR and two RL algorithms in the majority of time steps.

Hierarchical Maneuver Decision Method Based on PG-Option for UAV Pursuit-Evasion Game

Aiming at the autonomous decision-making problem in an Unmanned aerial vehicle (UAV) pursuit-evasion game, this paper proposes a hierarchical maneuver decision method based on the PG-option. Firstly, considering various situations of the relationship of both sides comprehensively, this paper designs four maneuver decision options: advantage game, quick escape, situation change and quick pursuit, and the four options are trained by Soft Actor-Critic (SAC) to obtain the corresponding meta-policy. In addition, to avoid high dimensions in the state space in the hierarchical model, this paper combines the policy gradient (PG) algorithm with the traditional hierarchical reinforcement learning algorithm based on the option. The PG algorithm is used to train the policy selector as the top-level strategy. Finally, to solve the problem of frequent switching of meta-policies, this paper sets the delay selection of the policy selector and introduces the expert experience to design the termination function of the meta-policies, which improves the flexibility of switching policies. Simulation experiments show that the PG-option algorithm has a good effect on UAV pursuit-evasion game and adapts to various environments by switching corresponding meta-policies according to current situation.

Three-Stage Stackelberg Game Enabled Clustered Federated Learning in Heterogeneous UAV Swarms

In the past decade, the unmanned aerial vehicles (UAVs) swarm has become the disruptive force reshaping our lives and work. In particular, advances in artificial intelligence have allowed multiple UAVs to coordinate their operations and work together to accomplish various complex tasks, one of which is Federated Learning (FL). As a promising distributed learning paradigm, FL can be adopted well with the limited resources and dynamic network topology of UAV swarms. However, the current FL's training process relies on homogeneous data paradigms, which require distributed UAVs to hold the same structure data. This ideal hypothesis is can not apply to the heterogeneous UAV swarms. To tackle this problem, in this paper, we design a clustered federated learning (CFL) architecture, in which we cluster UAV swarms based on the similarities between the participants' optimization directions. Then, we formulate the model trading among model owners, cluster heads, and UAV workers as a three-stage Stackelberg game to optimize the allocation of the limited resources. We design a hierarchical reinforcement learning algorithm to search for the Stackelberg equilibrium under the clustered federated learning system. The performance evaluation demonstrates the uniqueness and stability of the proposed three-stage master-slave game under the clustered framework, as well as the convergence and effectiveness of the reinforcement learning algorithm.

HAVEN: Hierarchical Cooperative Multi-Agent Reinforcement Learning with Dual Coordination Mechanism

Recently, some challenging tasks in multi-agent systems have been solved by some hierarchical reinforcement learning methods. Inspired by the intra-level and inter-level coordination in the human nervous system, we propose a novel value decomposition framework HAVEN based on hierarchical reinforcement learning for fully cooperative multi-agent problems. To address the instability arising from the concurrent optimization of policies between various levels and agents, we introduce the dual coordination mechanism of inter-level and inter-agent strategies by designing reward functions in a two-level hierarchy. HAVEN does not require domain knowledge and pre-training, and can be applied to any value decomposition variant. Our method achieves desirable results on different decentralized partially observable Markov decision process domains and outperforms other popular multi-agent hierarchical reinforcement learning algorithms.

End-to-End Entity Linking with Hierarchical Reinforcement Learning

Entity linking (EL) is the task of linking the text segments to the referring entities in the knowledge graph, typically decomposed into mention detection, and entity disambiguation. Compared to traditional methods treating the two tasks separately, recent end-to-end entity linking methods exploit the mutual dependency between mentions and entities to achieve better performance. However, existing end-to-end EL methods have problems utilizing the dependency of mentions and entities in the task. To this end, we propose to model the EL task as a hierarchical decision-making process and design a hierarchical reinforcement learning algorithm to solve the problem. We conduct extensive experiments to show that the proposed method achieves state-of-the-art performance in several EL benchmark datasets. Our code is publicly available at https://github.com/lhlclhl/he2eel.

Generating Function Reallocation to Handle Contingencies in Human–Robot Teaming Missions: The Cases in Lunar Surface Transportation

On lunar missions, efficient and safe transportation of human–robot systems is essential for the success of human exploration and scientific endeavors. Given the fact that transportation constructs bottlenecks for numerous typical lunar missions, it is appealing to investigate what function allocation strategies can generate optimal task implementation paths for robots with low-human workloads when the situation changes. Thus, this paper presents a novel approach to dynamic human–robot function allocation explicitly designed for team transportation in lunar missions. The proposed dynamic allocation framework aims to optimize human–robot collaboration by responding to existing and potential contingencies. First, a fitness concept model is designed to quantify the factors that motivate the functional adaptation of each agent in dynamic lunar mission scenarios. A hierarchical reinforcement learning (HRL) algorithm with two layers is then employed for decision-making and optimization of human–robot function allocation. Finally, the validity of the framework and algorithm proposed is validated by a series of human–robot function allocation experiments on a simulated environment that mimics lunar transportation scenarios, and is compared with the performance of other algorithms. In the future, path-planning algorithms can be incorporated into the proposed framework to improve the adaptability and efficiency of the human–robot function allocation in lunar missions.

Hierarchical multi-agent reinforcement learning for repair crews dispatch control towards multi-energy microgrid resilience

Extreme events are greatly impacting the normal operations of microgrids, which can lead to severe outages and affect the continuous supply of energy to customers, incurring substantial restoration costs. Repair crews (RCs) are regarded as crucial resources to provide system resilience owing to their mobility and flexibility characteristics in handling both transportation and energy systems. Nevertheless, effectively coordinating the dispatch of RCs towards system resilience is a complex decision-making problem, especially in the context of a multi-energy microgrid (MEMG) with enormous dynamics and uncertainties. To this end, this paper formulates the dispatch problem of RCs in a coupled transportation and power-gas network as a decentralized partially observable Markov decision process (Dec-POMDP). To solve this Dec-POMDP, a hierarchical multi-agent reinforcement learning (MARL) algorithm is proposed by featuring a two-level framework, where the high-level action is used for switching decision-making between transportation and power-gas networks, and the lower-level action constructed via the multi-agent proximal policy optimization (MAPPO) algorithm is used to compute the routing and repairing decisions of RCs in the transportation and power-gas networks, respectively. The proposed algorithm also introduces an abstracted critic network by integrating the load restoration status, which captures the system dynamics and stabilizes the training performance with privacy protection. Extensive case studies are evaluated on a coupled 6-bus power and 6-bus gas network integrated with a 9-node 12-edge transportation network. The proposed algorithm outperforms the conventional MARL algorithms in terms of policy quality, learning stability, and computational performance. Furthermore, the dispatch strategies of RCs are analyzed and their corresponding benefits for load restoration are also evaluated. Finally, the scalability of the proposed method is also investigated for a larger 33-bus power and 15-bus gas network integrated with an 18-node 27-edge transportation network.

Hierarchical Reinforcement Learning for Multi-Objective Real-Time Flexible Scheduling in a Smart Shop Floor

With the development of intelligent manufacturing, machine tools are considered the “mothership” of the equipment manufacturing industry, and the associated processing workshops are becoming more high-end, flexible, intelligent, and green. As the core of manufacturing management in a smart shop floor, research into the multi-objective dynamic flexible job shop scheduling problem (MODFJSP) focuses on optimizing scheduling decisions in real time according to changes in the production environment. In this paper, hierarchical reinforcement learning (HRL) is proposed to solve the MODFJSP considering random job arrival, with a focus on achieving the two practical goals of minimizing penalties for earliness and tardiness and reducing total machine load. A two-layer hierarchical architecture is proposed, namely the combination of a double deep Q-network (DDQN) and a dueling DDQN (DDDQN), and state features, actions, and external and internal rewards are designed. Meanwhile, a personal computer-based interaction feature is designed to integrate subjective decision information into the real-time optimization of HRL to obtain a satisfactory compromise. In addition, the proposed HRL framework is applied to multi-objective real-time flexible scheduling in a smart gear production workshop, and the experimental results show that the proposed HRL algorithm outperforms other reinforcement learning (RL) algorithms, metaheuristics, and heuristics in terms of solution quality and generalization and has the added benefit of real-time characteristics.

Developing Cooperative Policies for Multi-Stage Reinforcement Learning Tasks

Many hierarchical reinforcement learning algorithms utilise a series of independent skills as a basis to solve tasks at a higher level of reasoning. These algorithms don’t consider the value of using skills that are cooperative instead of independent. This paper proposes the Cooperative Consecutive Policies (CCP) method of enabling consecutive agents to cooperatively solve long time horizon multi-stage tasks. This method is achieved by modifying the policy of each agent to maximise both the current and next agent’s critic. Cooperatively maximising critics allows each agent to take actions that are beneficial for its task as well as subsequent tasks. Using this method in a multi-room maze domain and a peg in hole manipulation domain, the cooperative policies were able to outperform a set of naive policies, a single agent trained across the entire domain, as well as another sequential HRL algorithm.

Edge-Based Video Surveillance With Graph-Assisted Reinforcement Learning in Smart Construction

The smart construction site is developing rapidly with the intelligentization of industrial management. Intelligent devices are being widely deployed in construction industry to support artificial intelligence applications. Video surveillance is a core function of smart construction, which demands both high accuracy and low latency. The challenge is that the computation and networking resources in a construction site are often limited, and the inefficient scheduling policies create congestions in the network and bring additional delay that is unbearable to realtime surveillance. Adaptive video configuration and edge computing have been proposed to improve accuracy and reduce latency with limited resources. However, optimizing the video configuration and task scheduling in edge computing involves several factors that often interfere with each other, which significantly decreases the performance of video surveillance. In this article, we present an edge-based solution of video surveillance in the smart construction site assisted by a graph neural network. It leverages the distributed computing model to realize flexible allocation of resources. A graph-assisted hierarchical reinforcement learning algorithm is developed to illustrate the feature of the mobile-edge network and optimize the scheduling policy by the Deep- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula> Network. We implement and test the proposed solution in the commercial residential buildings of a fortune global 500 real estate company and observe that the proposed algorithm is efficient to maintain a reliable accuracy and keep lower delay. We further conduct a case study to demonstrate the superiority of the proposed solution by comparing it with traditional mechanisms.

Hierarchical Multihop Reasoning on Knowledge Graphs

Multihop knowledge reasoning aims to find missing entities for incomplete triples by finding paths on knowledge graphs. It is a fundamental and important task. In this article, we devise a hierarchical reinforcement learning algorithm to model the reasoning process more effectively. Unlike existing methods directly reason on entities and relations, we adopt a high-level reasoning layer to deal with abstract concepts, which guides the reasoning process conducted at the low level for concrete entities and relations. Our approach yields competitive results on link prediction on both NELL-995 and FB15k-237 datasets. The comparison to baselines also demonstrates the effectiveness of the hierarchical structure.

A Novel Hierarchical Soft Actor-Critic Algorithm for Multi-Logistics Robots Task Allocation

In intelligent unmanned warehouse goods-to-man systems, the allocation of tasks has an important influence on the efficiency because of the dynamic performance of AGV robots and orders. The paper presents a hierarchical Soft Actor-Critic algorithm to solve the dynamic scheduling problem of orders picking. The method proposed is based on the classic Soft Actor-Critic and hierarchical reinforcement learning algorithm. In this paper, the model is trained at different time scales by introducing sub-goals, with the top-level learning a policy and the bottom level learning a policy to achieve the sub-goals. The actor of the controller aims to maximize expected intrinsic reward while also maximizing entropy. That is, to succeed at the sub-goals while moving as randomly as possible. Finally, experimental results for simulation experiments in different scenes show that the method can make multi-logistics AGV robots work together and improves the reward in sparse environments about 2.61 times compared to the SAC algorithm.