Hierarchical Reinforcement Research Articles

Boosting Reinforcement Learning via Hierarchical Game Playing With State Relay.

Due to its wide application, deep reinforcement learning (DRL) has been extensively studied in the motion planning community in recent years. However, in the current DRL research, regardless of task completion, the state information of the agent will be reset afterward. This leads to a low sample utilization rate and hinders further explorations of the environment. Moreover, in the initial training stage, the agent has a weak learning ability in general, which affects the training efficiency in complex tasks. In this study, a new hierarchical reinforcement learning (HRL) framework dubbed hierarchical learning based on game playing with state relay (HGR) is proposed. In particular, we introduce an auxiliary penalty to regulate task difficulty, and one training mechanism, the state relay mechanism, is designed. The relay mechanism can make full use of the intermediate states of the agent and expand the environment exploration of low-level policy. Our algorithm can improve the sample utilization rate, reduce the sparse reward problem, and thereby enhance the training performance in complex environments. Simulation tests are carried out on two public experiment platforms, i.e., MazeBase and MuJoCo, to verify the effectiveness of the proposed method. The results show that HGR significantly benefits the reinforcement learning (RL) area.

Developing Driving Strategies Efficiently: A Skill-Based Hierarchical Reinforcement Learning Approach

Developing Driving Strategies Efficiently: A Skill-Based Hierarchical Reinforcement Learning Approach

Energy-Optimal Trajectory Planning for Near-Space Solar-Powered UAV Based on Hierarchical Reinforcement Learning

Energy-Optimal Trajectory Planning for Near-Space Solar-Powered UAV Based on Hierarchical Reinforcement Learning

Hierarchical Reinforcement Learning-Based Joint Allocation of Jamming Task and Power for Countering Networked Radar

Hierarchical Reinforcement Learning-Based Joint Allocation of Jamming Task and Power for Countering Networked Radar

Large-Scale Dynamic Scheduling for Flexible Job-Shop With Random Arrivals of New Jobs by Hierarchical Reinforcement Learning

As the intelligent manufacturing paradigm evolves, it is urgent to design a near real-time decision-making framework for handling the uncertainty and complexity of production line control. The dynamic flexible job shop scheduling problem (DFJSP) is frequently encountered in the manufacturing industry. However, it is still challenging to obtain high-quality schedules for DFJSP with dynamic job arrivals in real-time, especially facing thousands of operations from a large-scale scene with complex contexts in an assembly plant. This paper aims to propose a novel end-to-end hierarchical reinforcement learning framework for solving the large-scale DFJSP in near real-time. In the DFJSP, the processing information of newly arrived jobs is unknown in advance. Besides, two optimization tasks, including job operation selection and job-to-machine assignment, have to be handled, which means multiple actions must be controlled simultaneously. In our framework, a higher-level layer is designed to automatically divide the DFJSP into sub-problems, i.e., static FJSPs with different scales. And two lower-level layers are constructed to solve the sub-problems. In particular, one layer based on a graph neural network is in charge of sequencing job operations, and another layer based on a multi-layer perceptron is used to assign a machine to process the job operations. Numerical experiments, including offline training and online testing, are conducted on several instances with different scales. The results verify the superior performance of the proposed framework compared with existing dynamic scheduling methods, such as well-known dispatching rules and meta-heuristics.

Knowledge Graph-Enhanced Hierarchical Reinforcement Learning for Interactive and Explainable Recommendation

Knowledge Graph-Enhanced Hierarchical Reinforcement Learning for Interactive and Explainable Recommendation

Hierarchical Reinforcement Learning with Partner Modeling for Distributed Multi-agent Cooperation

Hierarchical Reinforcement Learning with Partner Modeling for Distributed Multi-agent Cooperation

Spatial memory-augmented visual navigation based on hierarchical deep reinforcement learning in unknown environments

Visual navigation in unknown environments poses significant challenges due to the presence of many obstacles and low-texture scenes. These factors may cause frequent collisions and tracking failure of feature-based visual Simultaneous Localization and Mapping (vSLAM). To avoid these issues, this paper proposes a spatial memory-augmented visual navigation system that combines a vSLAM module, a conventional global planner module, and a Hierarchical Reinforcement Learning (HRL)-based local planner module. Firstly, a real-time vSLAM named Salient-SLAM is proposed to augment the performance of visual navigation. Salient-SLAM creates a navigation mapping thread by combining a saliency prediction model to build a navigation map that categorizes environmental regions as occupied, explored, or noticeable. Spatial memory that contains spatial abstraction and saliency information of the environment can be further formed by encoding navigation maps, which helps the agent determine an optimal path towards its destination. An open-sourced saliency dataset is proposed to train the saliency prediction model by mimicking the visual attention mechanism. Secondly, a HRL method is proposed to automatically decompose local planning into a high-level policy selector and several low-level policies, where the latter produces actions to interact with the environment. We maximize entropy and minimize option correlation in learning low-level policies, aiming at acquiring diverse and independent behaviors. The simulation results show that the proposed HRL method outperforms competitive baselines by 6.29–10.85 % on Success Rate (SR) and 3.87–11.1 % on Success weighted by Path Length (SPL) metrics. By incorporating the spatial memory, SR, and SPL metrics can be augmented by an average of 9.85 % and 10.89 %, respectively.

Decomposing user-defined tasks in a reinforcement learning setup using TextWorld.

The current paper proposes a hierarchical reinforcement learning (HRL) method to decompose a complex task into simpler sub-tasks and leverage those to improve the training of an autonomous agent in a simulated environment. For practical reasons (i.e., illustrating purposes, easy implementation, user-friendly interface, and useful functionalities), we employ two Python frameworks called TextWorld and MiniGrid. MiniGrid functions as a 2D simulated representation of the real environment, while TextWorld functions as a high-level abstraction of this simulated environment. Training on this abstraction disentangles manipulation from navigation actions and allows us to design a dense reward function instead of a sparse reward function for the lower-level environment, which, as we show, improves the performance of training. Formal methods are utilized throughout the paper to establish that our algorithm is not prevented from deriving solutions.

Modular hierarchical reinforcement learning for multi-destination navigation in hybrid crowds

Real-world robot applications usually require navigating agents to face multiple destinations. Besides, the real-world crowded environments usually contain dynamic and static crowds that implicitly interact with each other during navigation. To address this challenging task, a novel modular hierarchical reinforcement learning (MHRL) method is developed in this paper. MHRL is composed of three modules, i.e., destination evaluation, policy switch, and motion network, which are designed exactly according to the three phases of solving the original navigation problem. First, the destination evaluation module rates all destinations and selects the one with the lowest cost. Subsequently, the policy switch module decides which motion network to be used according to the selected destination and the obstacle state. Finally, the selected motion network outputs the robot action. Owing to the complementary strengths of a variety of motion networks and the cooperation of modules in each layer, MHRL is able to deal with hybrid crowds effectively. Extensive simulation experiments demonstrate that MHRL achieves better performance than state-of-the-art methods.

Learning and Reusing Quadruped Robot Movement Skills from Biological Dogs for Higher-Level Tasks.

In the field of quadruped robots, the most classic motion control algorithm is based on model prediction control (MPC). However, this method poses challenges as it necessitates the precise construction of the robot's dynamics model, making it difficult to achieve agile movements similar to those of a biological dog. Due to these limitations, researchers are increasingly turning to model-free learning methods, which significantly reduce the difficulty of modeling and engineering debugging and simultaneously reduce real-time optimization computational burden. Inspired by the growth process of humans and animals, from learning to walk to fluent movements, this article proposes a hierarchical reinforcement learning framework for the motion controller to learn some higher-level tasks. First, some basic motion skills can be learned from motion data captured from a biological dog. Then, with these learned basic motion skills as a foundation, the quadruped robot can focus on learning higher-level tasks without starting from low-level kinematics, which saves redundant training time. By utilizing domain randomization techniques during the training process, the trained policy function can be directly transferred to a physical robot without modification, and the resulting controller can perform more biomimetic movements. By implementing the method proposed in this article, the agility and adaptability of the quadruped robot can be maximally utilized to achieve efficient operations in complex terrains.

Correction: Hierarchical reinforcement learning for kinematic control tasks with parameterized action spaces

Correction: Hierarchical reinforcement learning for kinematic control tasks with parameterized action spaces

A hierarchical reinforcement learning GPC for flexible operation of ultra-supercritical unit considering economy

With the growing proportion of renewable energy sources in power grid, the demands for deep peak shaving and rapid load regulation of ultra-supercritical (USC) unit to provide more flexible and stable power services is further enhanced. Simultaneously, the load tracking speed is no longer the only significant task in the flexible operation of USC unit, the operational economy is more and more important. To this end, a hierarchical reinforcement learning GPC control scheme with a better economy is proposed and apply to USC unit. Firstly, the model mismatch problem in control process is taken into account, and an integral mode is added into the GPC to eliminate the steady state error during the load regulation. Secondly, the hierarchical reinforcement learning GPC scheme integrated the GPC and twin delayed deep deterministic policy gradient algorithms is presented. The complex flexibility demand problem is decomposed into rapid load regulation and economic operation and then respectively solved in the GPC-based upper layer and reinforcement learning agent-based lower layer. Thirdly, under the flexibility demand, a multi-criteria optimization function including the load tracking cost, main steam throttling loss and coal consumption rate of unit is constructed. Then, via the accurate trajectory tracking by GPC and on-line control sequence optimization by reinforcement learning agent, the operational flexibility and economy of USC unit are simultaneously improved within safe boundaries. Finally, to verify the availability of the proposed hierarchical reinforcement learning GPC strategy, large-scale load variations and disturbance rejection ability tests cover the 30%–100 % rated load of a 1000 MW USC unit are carried out, and the load regulation performance is compared to other strategies. Results confirm that the designed hierarchical reinforcement learning GPC control scheme is profit for the flexible and economic operation capability promotion of USC unit.

An Actor-Critic Hierarchical Reinforcement Learning Model for Course Recommendation

Online learning platforms provide diverse course resources, but this often results in the issue of information overload. Learners always want to learn courses that are appropriate for their knowledge level and preferences quickly and accurately. Effective course recommendation plays a key role in helping learners select appropriate courses and improving the efficiency of online learning. However, when a user is enrolled in multiple courses, existing course recommendation methods face the challenge of accurately recommending the target course that is most relevant to the user because of the noise courses. In this paper, we propose a novel reinforcement learning model named Actor-Critic Hierarchical Reinforcement Learning (ACHRL). The model incorporates the actor-critic method to construct the profile reviser. This can remove noise courses and make personalized course recommendations effectively. Furthermore, we propose a policy gradient based on the temporal difference error to reduce the variance in the training process, to speed up the convergence of the model, and to improve the accuracy of the recommendation. We evaluate the proposed model using two real datasets, and the experimental results show that the proposed model significantly outperforms the existing recommendation models (improving 3.77% to 13.66% in terms of HR@5).

A Knowledge-Enhanced Hierarchical Reinforcement Learning-Based Dialogue System for Automatic Disease Diagnosis

Deep Reinforcement Learning is a key technology for the diagnosis-oriented medical dialogue system, determining the type of disease according to the patient’s utterances. The existing dialogue models for disease diagnosis cannot achieve good performance due to the large number of symptoms and diseases. In this paper, we propose a knowledge-enhanced hierarchical reinforcement learning model for strategy learning in the medical dialogue system for disease diagnosis. Our hierarchical strategy alleviates the problem of a large action space in reinforcement learning. In addition, the knowledge enhancement module integrates a learnable disease–symptom relationship matrix and medical knowledge graph into the hierarchical strategy for higher diagnosis success rate. Our proposed model has been proved to be effective on a medical dialogue dataset for automatic disease diagnosis.

Hierarchical Reinforcement Learning for Air Combat at DARPA's AlphaDogfight Trials

Autonomous control in high-dimensional, continuous state spaces is a persistent and important challenge in the fields of robotics and artificial intelligence. Because of high risk and complexity, the adoption of AI for autonomous combat systems has been a long-standing difficulty. In order to address these issues, DARPA's AlphaDogfight Trials (ADT) program sought to vet the feasibility of and increase trust in AI for autonomously piloting an F-16 in simulated air-to-air combat. Our submission to ADT solves the high-dimensional, continuous control problem using a novel hierarchical deep reinforcement learning approach consisting of a high-level policy selector and a set of separately trained low-level policies specialized for excelling in specific regions of the state space. Both levels of the hierarchy are trained using off-policy, maximum entropy methods with expert knowledge integrated through reward shaping. Our approach outperformed human expert pilots and achieved a second-place rank in the ADT championship event. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Impact Statement–</i> Significant performance milestones in reinforcement learning have been achieved in recent years, with autonomous agents demonstrating super-human performance across a wide variety of tasks. Before these algorithms can be extensively deployed in real-world defense applications, a greater level of trust must first be achieved. ADT was an important step towards developing the trust necessary to operationalize these algorithms, by demonstrating their effectiveness on a foundational yet relevant problem in a high-fidelity simulation environment. Developed for the program, our hierarchical reinforcement learning agent was designed alongside of and competed against active fighter pilots, and ultimately defeated a graduate of the United States Air Force's F-16 Weapons Instructor Course in match play.

Toward Scalable and Efficient Hierarchical Deep Reinforcement Learning for 5G RAN Slicing

Toward Scalable and Efficient Hierarchical Deep Reinforcement Learning for 5G RAN Slicing

Toward Symptom Assessment Guided Symptom Investigation and Disease Diagnosis

Automatic Disease Diagnosis (ADD) has gained immense popularity and demand over the past few years, and it is emerging as an effective diagnostic assistant to doctors. Diagnosis assistants assist clinicians in conducting a thorough symptom investigation and identifying possible diseases. Doctors correctly diagnose patients by observing only a few symptoms in most cases, even though the diagnosed disease has numerous symptoms. Also, some common symptoms, such as fever and headache, usually emerge due to other symptoms, which do not play a major role in identifying suffering diseases. In this work, we investigate the role of symptom importance in disease diagnosis through several feature engineering techniques and propose a novel symptom assessment incorporated symptom investigation and disease diagnosis (SA-SIDD) assistant using hierarchical reinforcement learning. The proposed SA-SIDD assistant first collects an adequate set of symptoms/sign information through conversing with users and then diagnoses a disease based on the extracted symptoms. We incorporated a symptom assessment module with the diagnosis framework that evaluates the relevance of current inspected symptom at each turn and reinforces the assistant to investigate distinctive and context-aligned symptoms using an assessment critic. The proposed methodology outperforms the state-of-the-art method, HRL, on two publicly available datasets, which firmly establishes the crucial role of symptom importance in disease diagnosis and the need for the proposed symptom assessment incorporated disease diagnosis framework. Furthermore, we have also conducted a human evaluation, revealing that the diagnosis method greatly enhances end-user satisfaction because of context-aligned relevant and minimal symptom investigation <xref ref-type="fn" rid="fn1" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">¹</xref> <fn id="fn1" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><label>¹</label> The code and data are available at <uri>https://github.com/NLP-RL/SA-SIDD</uri> </fn> .

Black-box attacks on dynamic graphs via adversarial topology perturbations

Research and analysis of attacks on dynamic graph is beneficial for information systems to investigate vulnerabilities and strength abilities in resisting malicious attacks. Existing attacks on dynamic graphs mainly focus on rewiring original graph structures, which are often infeasible in real-world scenarios. To address this issue, we adopt a novel strategy by injecting both fake nodes and links to attack dynamic graphs. Based on that, we present the first study on attacking dynamic graphs via adversarial topology perturbations in a restricted black-box setting, in which downstream graph learning tasks are unknown. Specifically, we first divide dynamic graph structure perturbations into three sub-tasks and transform them as a sequential decision making process. Then, we propose a hierarchical reinforcement learning based black-box attack (HRBBA) framework to model three sub-tasks as attack policies. In addition, an imperceptible perturbation constraint to guarantee the concealment of attacks is incorporated into HRBBA. Finally, HRBBA is optimized based on the actor-critic process. Extensive experiments on four real-world dynamic graphs show that the performance of diverse dynamic graph learning methods (victim methods) on tasks like link prediction, node classification and network clustering can be substantially degraded under HRBBA attack.

DisTop: Discovering a Topological Representation to Learn Diverse and Rewarding Skills

An efficient way for a deep reinforcement learning (DRL) agent to explore can be to learn a set of skills that achieves a uniform distribution of states. Following this, we introduce DisTop, a new model that simultaneously learns diverse skills and focuses on improving rewarding skills. DisTop progressively builds a discrete topology of the environment using an unsupervised contrastive loss, a growing network and a goal-conditioned policy. Using this topology, a state-independent hierarchical policy can select where the agent has to keep discovering skills in the state space and explicitly forget skills unrelated to tasks. In turn, the new set of visited states allows an improved learnt representation and the learning loop continues. Our experiments emphasize that DisTop is agnostic to the ground state representation and that the agent can discover the topology of its environment whether the states are high-dimensional binary data, images, or proprioceptive inputs. We demonstrate that this paradigm is competitive on MuJoCo benchmarks with state-of-the-art algorithms on both single-task dense rewards and diverse skill discovery. By combining these two aspects, we show that DisTop outperforms a state-of-the-art hierarchical reinforcement learning (HRL) algorithm when rewards are sparse. We believe DisTop opens new perspectives by showing that bottom-up skill discovery combined with representation learning can tackle different hard reward settings.