Hierarchical Reinforcement Learning Approach Research Articles

Towards efficient RAN slicing: A deep hierarchical reinforcement learning approach

Radio access network (RAN) slicing can significantly improve network flexibility and resource utilization efficiency. Generally, deep reinforcement learning (DRL) is a prevailing approach to implementing practicable resource management in RAN slicing. However, this demand-aware resource allocation suffers from long training time and slow execution efficiency. This can lead to the inability to quickly achieve the desired spectral efficiency and service level agreement satisfaction rate. To tackle this issue, we propose a novel RAN slicing resource allocation framework based on a deep hierarchical RL framework for efficient resource scheduling. Structurally, our framework consists of a policy selection network and a policy evaluation network. In particular, a newly built action acceleration unit can achieve quick reward accumulation, thus speeding up the optimal policy search. Extensive simulations show that our proposal has higher system utility and faster convergence compared to the state-of-the-art DRL algorithms.

Vehicle dispatching and routing of on-demand intercity ride-pooling services: A multi-agent hierarchical reinforcement learning approach

The integrated development of city clusters has given rise to an increasing demand for intercity travel. Intercity ride-pooling service exhibits considerable potential in upgrading traditional intercity bus services by implementing demand-responsive enhancements. Nevertheless, its online operations suffer the inherent complexities due to the coupling of vehicle resource allocation among cities and pooled-ride vehicle routing. To tackle these challenges, this study proposes a two-level framework designed to facilitate online fleet management. Specifically, a novel multi-agent feudal reinforcement learning model is proposed at the upper level of the framework to cooperatively assign idle vehicles to different intercity lines, while the lower level updates the routes of vehicles using an adaptive large neighborhood search heuristic. Numerical studies based on the realistic dataset of Xiamen and its surrounding cities in China show that the proposed framework effectively mitigates the supply and demand imbalances, and achieves significant improvement in both the average daily system profit and order fulfillment ratio.

Interactive learning for multi-finger dexterous hand: A model-free hierarchical deep reinforcement learning approach

When a multi-fingered dexterous hand interacts with the external environment, it encounters various challenges, including the utilization of complex control techniques and the intricate coordination of finger motion sequences. Previous studies have primarily concentrated on investigating the interaction between multi-fingered dexterous hands and external objects, usually using model-based control or model-free reinforcement learning techniques. However, during practical implementation, various constraining factors are encountered, such as intricate modeling and limited interaction capabilities. In practical scenarios, the utilization of multi-fingered dexterous hands is imperative for the swift and efficient execution of a wide range of interactive tasks, including but not limited to throwing a ball and playing rock-paper-scissors. These tasks require skilled manual dexterity to demonstrate both precise control and quick responsiveness. To tackle this issue, we propose a hierarchical control approach for multi-fingered dexterous hands with interactive functionalities, utilizing model-free deep reinforcement learning. The complex interaction task is decomposed into simple sub-tasks using hierarchical strategy and action primitive decomposition, which effectively reduces the complexity of the action space, and achieves the motion planning and end finger trajectory control of dexterous hand. In a simulated environment, the aforementioned method has successfully executed interactive tasks, including ball throwing and playing rock-paper-scissors. It achieved a maximum normalized reward of 0.83 and an 84 % success rate. These results are noteworthy in terms of both control accuracy and response speed. This study offers novel insights into the effective resolution of the intricate challenges associated with interactions involving multi-fingered dexterous hands and human-computer interaction.

Initial Task Allocation in Multi-Human Multi-Robot Teams: An Attention-Enhanced Hierarchical Reinforcement Learning Approach

Initial Task Allocation in Multi-Human Multi-Robot Teams: An Attention-Enhanced Hierarchical Reinforcement Learning Approach

Spatial-Temporal Interplay in Human Mobility: A Hierarchical Reinforcement Learning Approach with Hypergraph Representation

In the realm of human mobility, the decision-making process for selecting the next-visit location is intricately influenced by a trade-off between spatial and temporal constraints, which are reflective of individual needs and preferences. This trade-off, however, varies across individuals, making the modeling of these spatial-temporal dynamics a formidable challenge. To address the problem, in this work, we introduce the "Spatial-temporal Induced Hierarchical Reinforcement Learning" (STI-HRL) framework, for capturing the interplay between spatial and temporal factors in human mobility decision-making. Specifically, STI-HRL employs a two-tiered decision-making process: the low-level focuses on disentangling spatial and temporal preferences using dedicated agents, while the high-level integrates these considerations to finalize the decision. To complement the hierarchical decision setting, we construct a hypergraph to organize historical data, encapsulating the multi-aspect semantics of human mobility. We propose a cross-channel hypergraph embedding module to learn the representations as the states to facilitate the decision-making cycle. Our extensive experiments on two real-world datasets validate the superiority of STI-HRL over state-of-the-art methods in predicting users' next visits across various performance metrics.

Loyal wingman task execution for future aerial combat: A hierarchical prior-based reinforcement learning approach

In modern Beyond-Visual-Range (BVR) aerial combat, unmanned loyal wingmen are pivotal, yet their autonomous capabilities are limited. Our study introduces an advanced control algorithm based on hierarchical reinforcement learning to enhance these capabilities for critical missions like target search, positioning, and relay guidance. Structured on a dual-layer model, the algorithm’s lower layer manages basic aircraft maneuvers for optimal flight, while the upper layer processes battlefield dynamics, issuing precise navigational commands. This approach enables accurate navigation and effective reconnaissance for lead aircraft. Notably, our Hierarchical Prior-augmented Proximal Policy Optimization (HPE-PPO) algorithm employs a prior-based training, prior-free execution method, accelerating target positioning training and ensuring robust target reacquisition. This paper also improves missile relay guidance and promotes the effective guidance. By integrating this system with a human-piloted lead aircraft, this paper proposes a potent solution for cooperative aerial warfare. Rigorous experiments demonstrate enhanced survivability and efficiency of loyal wingmen, marking a significant contribution to Unmanned Aerial Vehicles (UAV) formation control research. This advancement is poised to drive substantial interest and progress in the related technological fields.

Efficient relation extraction via quantum reinforcement learning

Most existing relation extraction methods only determine the relation type after identifying all entities, thus not fully modeling the interaction between relation-type recognition and entity mention detection. This article introduces a novel paradigm for relation extraction by treating relevant entities as parameters of relations and harnessing the strong expressive capabilities and acceleration advantages of quantum computing to address the relation extraction task. In this article, we develop a quantum hierarchical reinforcement learning approach to enhance the interaction between relation-type recognition and entity mention detection. The entire relation extraction process is broken down into a hierarchical structure of two layers of quantum reinforcement learning strategies dedicated to relation detection and entity extraction, demonstrating greater feasibility and expressiveness, especially when dealing with superimposed relations. Our proposed method outperforms existing approaches through experimental evaluations on commonly used public datasets, mainly showcasing its significant advantages in extracting superimposed relationships.

Toward Fair Power Grid Control: A Hierarchical Multiobjective Reinforcement Learning Approach

Toward Fair Power Grid Control: A Hierarchical Multiobjective Reinforcement Learning Approach

Electric vehicle charging navigation strategy in coupled smart grid and transportation network: A hierarchical reinforcement learning approach

Most of the existing electric vehicle (EV) charging navigation methods do not simultaneously take into account the electric vehicle charging destination optimization and path planning. Moreover, they are unable to provide online real-time decision-making under a variety of uncertain factors. To address these problems, this paper first establishes a bilevel stochastic optimization model for EV charging navigation considering various uncertainties, and then proposes an EV charging navigation method based on the hierarchical enhanced deep Q network (HEDQN) to solve the above stochastic optimization model in real-time. The proposed HEDQN contains two enhanced deep Q networks, which are utilized to optimize the charging destination and charging route path of EVs, respectively. Finally, the proposed method is simulated and validated in two urban transportation networks. The simulation results demonstrate that compared with the Dijkstra shortest path algorithm, single-layer deep reinforcement learning algorithm, and traditional hierarchical deep reinforcement learning algorithm, the proposed HEDQN algorithm can effectively reduce the total charging cost of electric vehicles and realize online real-time charging navigation of electric vehicles, that shows excellent generalization ability and scalability.

Energy-efficient access point clustering and power allocation in cell-free massive MIMO networks: a hierarchical deep reinforcement learning approach

Cell-free massive multiple-input multiple-output (CF-mMIMO) has attracted considerable attention due to its potential for delivering high data rates and energy efficiency (EE). In this paper, we investigate the resource allocation of downlink in CF-mMIMO systems. A hierarchical depth deterministic strategy gradient (H-DDPG) framework is proposed to jointly optimize the access point (AP) clustering and power allocation. The framework uses two-layer control networks operating on different timescales to enhance EE of downlinks in CF-mMIMO systems by cooperatively optimizing AP clustering and power allocation. In this framework, the high-level processing of system-level problems, namely AP clustering, enhances the wireless network configuration by utilizing DDPG on the large timescale while meeting the minimum spectral efficiency (SE) constraints for each user. The low layer solves the link-level sub-problem, that is, power allocation, and reduces interference between APs and improves transmission performance by utilizing DDPG on a small timescale while meeting the maximum transmit power constraint of each AP. Two corresponding DDPG agents are trained separately, allowing them to learn from the environment and gradually improve their policies to maximize the system EE. Numerical results validate the effectiveness of the proposed algorithm in term of its convergence speed, SE, and EE.

Joint Optimization on Uplink OFDMA and MU-MIMO for IEEE 802.11ax: Deep Hierarchical Reinforcement Learning Approach

Joint Optimization on Uplink OFDMA and MU-MIMO for IEEE 802.11ax: Deep Hierarchical Reinforcement Learning Approach

Developing Driving Strategies Efficiently: A Skill-Based Hierarchical Reinforcement Learning Approach

Developing Driving Strategies Efficiently: A Skill-Based Hierarchical Reinforcement Learning Approach

Hierarchical Reinforcement Learning for Air Combat at DARPA's AlphaDogfight Trials

Autonomous control in high-dimensional, continuous state spaces is a persistent and important challenge in the fields of robotics and artificial intelligence. Because of high risk and complexity, the adoption of AI for autonomous combat systems has been a long-standing difficulty. In order to address these issues, DARPA's AlphaDogfight Trials (ADT) program sought to vet the feasibility of and increase trust in AI for autonomously piloting an F-16 in simulated air-to-air combat. Our submission to ADT solves the high-dimensional, continuous control problem using a novel hierarchical deep reinforcement learning approach consisting of a high-level policy selector and a set of separately trained low-level policies specialized for excelling in specific regions of the state space. Both levels of the hierarchy are trained using off-policy, maximum entropy methods with expert knowledge integrated through reward shaping. Our approach outperformed human expert pilots and achieved a second-place rank in the ADT championship event. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Impact Statement–</i> Significant performance milestones in reinforcement learning have been achieved in recent years, with autonomous agents demonstrating super-human performance across a wide variety of tasks. Before these algorithms can be extensively deployed in real-world defense applications, a greater level of trust must first be achieved. ADT was an important step towards developing the trust necessary to operationalize these algorithms, by demonstrating their effectiveness on a foundational yet relevant problem in a high-fidelity simulation environment. Developed for the program, our hierarchical reinforcement learning agent was designed alongside of and competed against active fighter pilots, and ultimately defeated a graduate of the United States Air Force's F-16 Weapons Instructor Course in match play.

When Moving Target Defense Meets Attack Prediction in Digital Twins: A Convolutional and Hierarchical Reinforcement Learning Approach

When Moving Target Defense Meets Attack Prediction in Digital Twins: A Convolutional and Hierarchical Reinforcement Learning Approach

A Hierarchical Deep Learning Approach for Optimizing CCA Threshold and Transmit Power in Wi-Fi Networks

The nodes, e.g., access points and clients, in current WiFi networks rely on carrier sense multiple access (CSMA) for channel access. This means they rely on a clear channel assessment (CCA) threshold to determine when it is safe to transmit a packet. In practice, this threshold is fixed, and nodes do not adapt their CCA threshold given varying channel conditions. To this end, this paper proposes and studies the use of deep reinforcement learning (DRL) to optimize the CCA threshold and transmit power of an AP. It allows an AP to learn the optimal policy for setting its CCA threshold and transmit power given its queue length and interference level. We outline a semi-Markov decision process (Semi-MDP) for the problem at hand and solve it using a decentralized deep hierarchical reinforcement learning (HRL) approach. We then evaluate its performance using traffic generated via a Poisson distribution and also from a trace file. The results show that when compared to two competing solutions, an AP with HRL reduces its average queue length by 24.4% and 52.4%, respectively.

Optimal continuous control of refrigerator for electricity cost minimization—Hierarchical reinforcement learning approach

A refrigerator is a commonly used household appliance; however, limited research has focused on optimizing temperature control policy with the consideration of Time-of-Use (ToU) electricity price. This paper introduces a novel framework that utilizes hierarchical reinforcement learning (HRL) to control the intensity of refrigerator motors. The objective is to achieve both temperature regulation and cost savings under ToU and stochastic usage patterns. The problem is tackled by two HRL agents. The high-level agent is responsible for determining temperature reference based on ToU pricing, while the low-level agent adjusts the motor intensity to meet this temperature reference. To tackle non-stationarity in HRL, the high-level agent employs hindsight action transition and reward function approximation, while the low-level agent employs hindsight goal transition. Through the experimental evaluation, we found that the proposed method outperforms the conventional control methods and standard reinforcement learning approaches. It achieves the lowest total costs, resulting in a significant cost reduction of 5%–24%.

Vision-Based Autonomous Driving: A Hierarchical Reinforcement Learning Approach

Human drivers have excellent perception and reaction abilities in complex environments such as dangerous highways, busy intersections, and harsh weather conditions. To achieve human-level driving performance, autonomous driving systems require powerful environmental perception systems and the ability to make accurate decisions in difficult situations, enabling vehicles to maintain smooth driving. However, making decisions based on visual perception is still a daunting challenge for autonomous vehicles, also current end-to-end networks and modular frameworks have limitations in effectively addressing perception, decision-making, and control, such as a lack of interpretability and weak generalization ability in complex environments. This paper proposes an elaborate modular pipeline for autonomous driving that effectively integrates semantic perception information, multi-level decision tasks, and control modules. The decision-making module comprehensively considers high-level maneuver selection and low-level motion control in both horizontal and vertical directions. The proposed MP framework is trained end-to-end by a novel hierarchical reinforcement learning method with a new action sampling mechanism. To adapt to the experimental scenarios, we use the CARLA simulation platform to collect data to evaluate the proposed autonomous driving framework and its new training method, considering various environmental factors such as sunny, dusk, and rainy weather. The results show that the framework exhibits smooth and effective driving strategies in different environments and can converge quickly and stably. The method improves learning efficiency and reduces unnecessary coupling and error propagation. Overall, the proposed framework and training method provide new ideas for improving existing autonomous driving systems.

On UAV Serving Node Deployment for Temporary Coverage in Forest Environment: A Hierarchical Deep Reinforcement Learning Approach

On UAV Serving Node Deployment for Temporary Coverage in Forest Environment: A Hierarchical Deep Reinforcement Learning Approach

Hierarchical Deep Reinforcement Learning With Experience Sharing for Metaverse in Education

Metaverse has gained increasing interest in education, with much of literature focusing on its great potential to enhance both individual and social aspects of learning. However, little work has been done to address the systems and technologies behind providing meaningful Metaverse learning. This article proposes a technical framework to address this research gap, where a hierarchical multiagent reinforcement learning approach with experience sharing is developed to augment the intelligence of nonplayer characters in Metaverse learning for personalization. The utility and benefits of the proposed framework and methodologies are demonstrated in Gridlock, a Metaverse learning game, as well as through extensive simulations.

Computational Offloading for MEC Networks with Energy Harvesting: A Hierarchical Multi-Agent Reinforcement Learning Approach

Multi-access edge computing (MEC) is a novel computing paradigm that leverages nearby MEC servers to augment the computational capabilities of users with limited computational resources. In this paper, we investigate the computational offloading problem in multi-user multi-server MEC systems with energy harvesting, aiming to minimize both system latency and energy consumption by optimizing task offload location selection and task offload ratio.We propose a hierarchical computational offloading strategy based on multi-agent reinforcement learning (MARL). The proposed strategy decomposes the computational offloading problem into two sub-problems: a high-level task offloading location selection problem and a low-level task offloading ratio problem. The complexity of the problem is reduced by decoupling. To address these sub-problems, we propose a computational offloading framework based on multi-agent proximal policy optimization (MAPPO), where each agent generates actions based on its observed private state to avoid the problem of action space explosion due to the increasing number of user devices. Simulation results show that the proposed HDMAPPO strategy outperforms other baseline algorithms in terms of average task latency, energy consumption, and discard rate.