Hierarchical Policy Research Articles

E2HRL: An Energy-efficient Hardware Accelerator for Hierarchical Deep Reinforcement Learning

Recently, Reinforcement Learning (RL) has shown great performance in solving sequential decision-making and control in dynamic environment problems. Despite its achievements, deploying Deep Neural Network (DNN)-based RL is expensive in terms of time and power due to the large number of episodes required to train agents with high dimensional image representations. Additionally, at the interference the large energy footprint of deep neural networks can be a major drawback. Embedded edge devices as the main platform for deploying RL applications are intrinsically resource-constrained and deploying deep neural network-based RL on them is a challenging task. As a result, reducing the number of actions taken by the RL agent to learn desired policy, along with the energy-efficient deployment of RL, is crucial. In this article, we propose Energy Efficient Hierarchical Reinforcement Learning (E2HRL), which is a scalable hardware architecture for RL applications. E2HRL utilizes a cross-layer design methodology for achieving better energy efficiency, smaller model size, higher accuracy, and system integration at the software and hardware layers. Our proposed model for RL agent is designed based on the learning hierarchical policies, which makes the network architecture more efficient for implementation on mobile devices. We evaluated our model in three different RL environments with different level of complexity. Simulation results with our analysis illustrate that hierarchical policy learning with several levels of control improves RL agents training efficiency and the agent learns the desired policy faster compared to a non-hierarchical model. This improvement is specifically more observable as the environment or the task becomes more complex with multiple objective subgoals. We tested our model with different hyperparameters to achieve the maximum reward by the RL agent while minimizing the model size, parameters, and required number of operations. E2HRL model enables efficient deployment of RL agent on resource-constraint-embedded devices with the proposed custom hardware architecture that is scalable and fully parameterized with respect to the number of input channels, filter size, and depth. The number of processing engines (PE) in the proposed hardware can vary between 1 to 8, which provides the flexibility of tradeoff of different factors such as latency, throughput, power, and energy efficiency. By performing a systematic hardware parameter analysis and design space exploration, we implemented the most energy-efficient hardware architectures of E2HRL on Xilinx Artix-7 FPGA and NVIDIA Jetson TX2. Comparing the implementation results shows Jetson TX2 boards achieve 0.1 ∼ 1.3 GOP/S/W energy efficiency while Artix-7 FPGA achieves 1.1 ∼ 11.4 GOP/S/W, which denotes 8.8× ∼ 11× better energy efficiency of E2HRL when model is implemented on FPGA. Additionally, compared to similar works our design shows better performance and energy efficiency.

PG-CODE: Latent dirichlet allocation embedded policy knowledge graph for government department coordination

Government policy-group integration and policy-chain inference are significant to the execution of strategies in current Chinese society. Specifically, the coordination of hierarchical policies implemented among government departments is one of the key challenges to rural revitalization. In recent years, various well-established quantitative methods have been proposed to evaluate policy coordination, but the majority of these relied on manual analysis, which can lead to subjective results. Thus, in this paper, a novel approach called "policy knowledge graph for the coordination among the government departments" (PG-CODE) is proposed, which incorporates topic modeling into policy knowledge graphs. Similar to a knowledge graph, a policy knowledge graph uses a graph-structured data model to integrate policy discourse. With latent Dirichlet allocation embedding, a policy knowledge graph could capture the underlying topics of the policies. Furthermore, coordination strength and topic diffusion among hierarchical departments could be inferred from the PG-CODE, as it can provide a better representation of coordination within the policy space. We implemented and evaluated the PG-CODE in the field of rural innovation and entrepreneurship policy, and the results effectively demonstrate improved coordination among departments.

Hierarchical reinforcement learning for automatic disease diagnosis.

Disease diagnosis-oriented dialog system models the interactive consultation procedure as the Markov decision process, and reinforcement learning algorithms are used to solve the problem. Existing approaches usually employ a flat policy structure that treat all symptoms and diseases equally for action making. This strategy works well in a simple scenario when the action space is small; however, its efficiency will be challenged in the real environment. Inspired by the offline consultation process, we propose to integrate a hierarchical policy structure of two levels into the dialog system for policy learning. The high-level policy consists of a master model that is responsible for triggering a low-level model, the low-level policy consists of several symptom checkers and a disease classifier. The proposed policy structure is capable to deal with diagnosis problem including large number of diseases and symptoms. Experimental results on three real-world datasets and a synthetic dataset demonstrate that our hierarchical framework achieves higher accuracy and symptom recall in disease diagnosis compared with existing systems. We construct a benchmark including datasets and implementation of existing algorithms to encourage follow-up researches. The code and data are available from https://github.com/FudanDISC/DISCOpen-MedBox-DialoDiagnosis. Supplementary data are available at Bioinformatics online.

Hierarchical policy with deep-reinforcement learning for nonprehensile multiobject rearrangement

Nonprehensile multiobject rearrangement is the robotic task of planning feasible paths and transferring multiple objects to their predefined target poses without grasping. It must consider how each object reaches the target and the order in which objects move, considerably increasing the complexity of the problem. Thus, we propose a hierarchical policy for nonprehensile multiobject rearrangement based on deep-reinforcement learning. We use imitation learning and reinforcement learning to train a rollout policy. In a high-level policy, the policy network directs the Monte Carlo tree search algorithm to efficiently seek the ideal rearrangement sequence for several items. In a low-level policy, the robot plans the paths according to the order of path primitives and manipulates the objects to approach the target poses one by one. Our experiments show that the proposed method has a higher success rate, fewer steps, and shorter path length than the state-of-the-art methods.

Muscle‐driven virtual human motion generation approach based on deep reinforcement learning

AbstractWe propose a muscle‐driven motion generation approach to realize virtual human motion with user interaction and higher fidelity, which can address the problem that the joint‐driven fails to reflect the motion process of the human body. First, a simplified virtual human musculoskeletal model is built based on human biomechanics. Then, a hierarchical policy learning framework is constructed including motion tracking layer, SPD controller and muscle control layer. The motion tracking layer is responsible for mimicking reference motion and completing control command, using proximal policy optimization to train the policy; the muscle control layer is aimed to minimize muscle energy consumption and train the policy based on supervised learning; the SPD controller acts as a link between the two layers. At the same time, we integrate the curriculum learning to improve the efficiency and success rate of policy training. Simulation experiments show that the proposed approach can use motion capture data and pose estimation data as reference motions to generate better and more adaptable motions. Furthermore, the virtual human has the ability to respond to the user control command during the motion, and can complete the target task successfully.

HLifeRL: A hierarchical lifelong reinforcement learning framework

Deep reinforcement learning research in a single-task environment has made remarkable achievements. However, it is often plagued by catastrophic forgetting, prohibitively low sample efficiency and lack of scalability problems when facing multi-task environment. To solve these issues, a Hierarchical Lifelong Reinforcement Learning framework (HLifeRL) is proposed to enhance the ability of agents to deal with a sequence of tasks in the way of skill discovery (we treat option as low-level skill in this paper) and hierarchical policy. HLifeRL can automatically extract task-related knowledge without any human intervention or priori knowledge. Moreover, with the help of a scalable library and the master policy, we can flexibly combine various skills to complete multiple tasks in the form of call-and-return. The experimental results show that HLifeRL can accelerate the speed of single-task training and deliver remarkable stability along with scalability in a lifelong setting environment.

Network Policy Enforcement With Commodity Multiqueue NICs for Multitenant Data Centers

Data centers are the fundamental component in the Internet of Things (IoT) system architecture. Data center servers where IoT services are co-located require hierarchical network policy enforcement to ensure fair bandwidth sharing among tenants and to prioritize latency-sensitive traffic within a tenant simultaneously. Meanwhile, emerging network interface cards (NICs) in servers make use of multiple hardware queues to drive increasing line rates. Unfortunately, multiqueue NICs make it hard to enforce hierarchical policies because the NIC packet scheduler dequeues packets in a static round-robin (RR) fashion for per-flow fairness. In this article, we enable hierarchical network policy enforcement with existing commodity multiqueue NICs. We design TONIC, a multiqueue NIC packet scheduling solution that approximates hierarchical packet scheduling by manipulating the packet dequeueing sequence of the NIC scheduler through dynamic packet enqueueing decisions. Specifically, TONIC leverages multiple hardware queues and the double-ended queue structure of qdiscs to express different tenant weights and application priorities without hardware modifications. We implement a TONIC prototype as a Linux kernel module and evaluate it on a testbed with commodity multiqueue NICs. Our experiment results show that TONIC can enforce hierarchical policies consisting of weighted fair sharing and traffic prioritization while maintaining robustness to various network conditions.

An Unequal Ethiopia in an Unequal World: Global and Domestic Hierarchies in Afäwärḳ Gäbrä-Iyyäsus’s and Käbbädä Mikael’s Political Thought (1908 and 1949)

ABSTRACT The Ethiopian empire retained its political independence through the European Scramble for Africa. The imperial elites oversaw the transformation of the empire into a territorially-bounded state, part of an international system of states regulated by international law and by international institutions such as the League of Nations, and later the UN. Ethiopian intellectuals were keenly aware that Ethiopia had joined this international system from a subordinated position and that its sovereignty remained at risk. The struggle for sovereignty was fought not only at a diplomatic level, but also at a narrative level. Afäwärḳ Gäbrä-Iyyäsus’s 1908 Traveller’s Guide to Abyssinia and Käbbädä Mikael’s 1949 Ethiopia and Western Civilisation pushed back against the European depiction of Ethiopia as intrinsically inferior and intrinsically unable to develop. Both Afäwärḳ and Käbbädä rejected the rigid determinism of stagist models of development, and argued that Ethiopia and Europe were natural allies by virtue of their shared Christian heritage. Global power hierarchies rigidified Ethiopia’s domestic power hierarchies. The article shows how the way in which Afäwärḳ and Käbbädä defended Ethiopia’s place in an unequal world had important consequences on their vision of domestic nation building, resulting in hierarchical assimilationist policies that marginalised Ethiopia’s non-Christian citizens.

A SYN flooding attack detection approach with hierarchical policies based on self‐information

The SYN flooding attack is widely used in cyber attacks because it paralyzes the network by causing the system and bandwidth resources to be exhausted. This paper proposed a self-information approach for detecting the SYN flooding attack and provided a detection algorithm with a hierarchical policy on a detection time domain. Compared with other detection methods of entropy measurement, the proposed approach is more efficient in detecting the SYN flooding attack, providing low misjudgment, hierarchical detection policy, and low time complexity. Furthermore, we proposed a detection algorithm with limiting system resources. Thus, the time complexity of our approach is only (log n) with lower time complexity and misjudgment rate than other approaches. Therefore, the approach can detect the denial-of-service/distributed denial-of-service attacks and prevent SYN flooding attacks.

Shaping Individualized Impedance Landscapes for Gait Training via Reinforcement Learning

Assist-as-needed (AAN) control aims at promoting therapeutic outcomes in robot-assisted rehabilitation by encouraging patients' active participation. Impedance control is used by most AAN controllers to create a compliant force field around a target motion to ensure tracking accuracy while allowing moderate kinematic errors. However, since the parameters governing the shape of the force field are often tuned manually or adapted online based on simplistic assumptions about subjects' learning abilities, the effectiveness of conventional AAN controllers may be limited. In this work, we propose a novel adaptive AAN controller that is capable of autonomously reshaping the force field in a phase-dependent manner according to each individual's motor abilities and task requirements. The proposed controller consists of a modified Policy Improvement with Path Integral algorithm, a model-free, sampling-based reinforcement learning method that learns a subject-specific impedance landscape in real-time, and a hierarchical policy parameter evaluation structure that embeds the AAN paradigm by specifying performance-driven learning goals. The adaptability of the proposed control strategy to subjects' motor responses and its ability to promote short-term motor adaptations are experimentally validated through treadmill training sessions with able-bodied subjects who learned altered gait patterns with the assistance of a powered ankle-foot orthosis.

Hierarchical Landmark Policy Optimization for Visual Indoor Navigation

In this paper, we study the problem of visual indoor navigation to an object that is defined by its semantic category. Recent works have shown significant achievements in the end-to-end reinforcement learning approach and modular systems. However, both approaches need a big step forward to be robust and practically applicable. To solve the problem of insufficient exploration of the scenes and make exploration more semantically meaningful, we extend standard task formulation and give the agent easily accessible landmarks in the form of the room locations and those types. The availability of landmarks allows the agent to build a hierarchical policy structure and achieve a success rate of 63% on validation scenes in a photo-realistic Habitat simulator. In a hierarchy, a low level consists of separately trained RL skills and a high level deterministic policy, which decides which skill is needed at the moment. Also, in this paper, we show the possibility of transferring a trained policy to a real robot. After a bit of training on the reconstructed real scene, the robot shows up to 79% SPL when solving the task of navigating to an arbitrary object.

Active Inference Integrated With Imitation Learning for Autonomous Driving

Classical imitation learning methods suffer substantially from the learning hierarchical policies when the imitative agent faces an unobserved state by the expert agent. To address these drawbacks, we propose an online active learning through active inference approach that encodes the expert’s demonstrations based on observation-action to improve the learner’s future motion prediction. For this purpose, we provide a switching Dynamic Bayesian Network based on the dynamic interaction between the expert agent and another object in its surrounding as a reference model, which we exploit to initialize an incremental probabilistic learning model. This learning model grows and matures based on the free-energy formulation and message passing of active inference dynamically at discrete and continuous levels in an online active learning phase. In this scheme, generalized states of the learning world are represented as distance-vector, where it is the learner’s observation concerning its interaction with a moving object. Considering the distance vector entail intentions, it enables action prediction evaluation in a prospective sense. We illustrate these points using simulations of driving intelligent agents. The learning agent is trained by using long-term predictions from the generative learning model to reproduce the expert’s motion while learning how to select a suitable action through new experiences. Our results affirm that a Dynamic Bayesian optimal approach provides a principled framework and outperforms conventional reinforcement learning methods. Furthermore, it endorses the general formulation of action prediction as active inference.

Dynamic Adaptation Method of Business Process Based on Hierarchical Feature Model

With the continuous complexity and frequent changes in business application scenarios, companies urgently need to establish a flexible business process management mechanism that includes dynamic rules, in which dynamic adaptation methods of business processes play a vital role. Aiming at the problem that the current methods only use the preset process template and the decision-making database, it cannot respond quickly to business changes and reconfigure the business process. This research proposes a dynamic adaptation method of business process based on the hierarchical feature model, builds a hierarchical feature model of complex processes, then establishes a hierarchical business policy set to achieve an agile response to business emergencies. By constructing a mapping model, the feature model is associated with the BPMN model to realize the rapid execution of the reconfiguration process model. The feasibility and effectiveness of the proposed method are verified by process examples and the developed business process dynamic adaptation tool.

Efficient hierarchical policy network with fuzzy rules

Efficient hierarchical policy network with fuzzy rules

Hierarchical Reinforcement Learning

Hierarchical Reinforcement Learning (HRL) enables autonomous decomposition of challenging long-horizon decision-making tasks into simpler subtasks. During the past years, the landscape of HRL research has grown profoundly, resulting in copious approaches. A comprehensive overview of this vast landscape is necessary to study HRL in an organized manner. We provide a survey of the diverse HRL approaches concerning the challenges of learning hierarchical policies, subtask discovery, transfer learning, and multi-agent learning using HRL. The survey is presented according to a novel taxonomy of the approaches. Based on the survey, a set of important open problems is proposed to motivate the future research in HRL. Furthermore, we outline a few suitable task domains for evaluating the HRL approaches and a few interesting examples of the practical applications of HRL in the Supplementary Material.

Intrinsically Motivated Hierarchical Policy Learning in Multiobjective Markov Decision Processes

The multiobjective Markov decision processes (MOMDPs) are sequential decision-making problems that involve multiple conflicting reward functions that cannot be optimized simultaneously without a compromise. This type of problem cannot be solved by a single optimal policy as in the conventional case. Alternatively, multiobjective reinforcement learning (RL) methods evolve a coverage set of optimal policies that can satisfy all possible preferences in solving the problem. However, many of these methods cannot generalize their coverage sets to work in the nonstationary environments. In these environments, the parameters of the state transition and reward distribution vary over time. This limitation results in significant performance degradation for the evolved policy sets. In order to overcome this limitation, there is a need to learn a generic skillset that can bootstrap the evolution of the policy coverage set for each shift in the environment dynamics, therefore, it can facilitate a continuous learning process. In this article, intrinsically motivated RL (IMRL) has been successfully deployed to evolve generic skillsets for learning hierarchical policy to solve the MOMDPs. We propose a novel dual-phase IMRL method to address this limitation. In the first phase, a generic set of skills is learned, while in the second phase, this set is used to bootstrap policy coverage sets for each shift in the environment dynamics. We show experimentally that the proposed method significantly outperforms the state-of-the-art multiobjective reinforcement methods on a dynamic robotics environment.

Layered Relative Entropy Policy Search

Many reinforcement learning problems are hierarchical in nature. Exploiting this property can ease the learning process. In this paper, a hierarchical policy search method based on clustering is presented. We use a hierarchical policy which is composed of a high-level gating policy and a set of low-level sub-policies. Depending on the observed state, the gating policy chooses one of the sub-policies for action selection. The gating policy and each sub-policy is provided with a learning method to update their parameters. If a task is truly hierarchical, its state–action space must have meaningful clusters. We cluster the observed samples and use each cluster to update one sub-policy. This way, each sub-policy is adapted to some portion of the state–action space. Using the updated sub-policy probability density functions and observed samples, the gating policy is updated. We evaluate our method on three multimodal tasks as well as a simulated and real robotic manipulation task. Our experiments show that our method can discover versatile sub-policies for multimodal tasks and the manipulation task. Moreover, we point out the errors in some of the equations of Hierarchical Relative Entropy Policy Search paper and provide the necessary corrections.

Playing Against the Board: Rolling Horizon Evolutionary Algorithms Against Pandemic

Competitive board games have provided a rich and diverse testbed for artificial intelligence. This paper contends that collaborative board games pose a different challenge to artificial intelligence as it must balance short-term risk mitigation with long-term winning strategies. Collaborative board games task all players to coordinate their different powers or pool their resources to overcome an escalating challenge posed by the board and a stochastic ruleset. This paper focuses on the exemplary collaborative board game Pandemic and presents a rolling horizon evolutionary algorithm designed specifically for this game. The complex way in which the Pandemic game state changes in a stochastic but predictable way required a number of specially designed forward models, macro-action representations for decision-making, and repair functions for the genetic operations of the evolutionary algorithm. Variants of the algorithm which explore optimistic versus pessimistic game state evaluations, different mutation rates and event horizons are compared against a baseline hierarchical policy agent. Results show that an evolutionary approach via short-horizon rollouts can better account for the future dangers that the board may introduce, and guard against them. Results highlight the types of challenges that collaborative board games pose to artificial intelligence, especially for handling multi-player collaboration interactions.

Efficient Robotic Object Search Via HIEM: Hierarchical Policy Learning With Intrinsic-Extrinsic Modeling

Despite the significant success at enabling robots with autonomous behaviors makes deep reinforcement learning a promising approach for robotic object search task, the deep reinforcement learning approach severely suffers from the nature sparse reward setting of the task. To tackle this challenge, we present a novel policy learning paradigm for the object search task, based on hierarchical and interpretable modeling with an intrinsic-extrinsic reward setting. More specifically, we explore the environment efficiently through a proxy low-level policy which is driven by the intrinsic rewarding sub-goals. We further learn our hierarchical policy from the efficient exploration experience where we optimize both of our high-level and low-level policies towards the extrinsic rewarding goal to perform the object search task well. Experiments conducted on the House3D environment validate and show that the robot, trained with our model, can perform the object search task in a more optimal and interpretable way.

Multiagent Hierarchical Cognition Difference Policy for Multiagent Cooperation

Multiagent cooperation is one of the most attractive research fields in multiagent systems. There are many attempts made by researchers in this field to promote cooperation behavior. However, several issues still exist, such as complex interactions among different groups of agents, redundant communication contents of irrelevant agents, which prevents the learning and convergence of agent cooperation behaviors. To address the limitations above, a novel method called multiagent hierarchical cognition difference policy (MA-HCDP) is proposed in this paper. It includes a hierarchical group network (HGN), a cognition difference network (CDN), and a soft communication network (SCN). HGN is designed to distinguish different underlying information of diverse groups’ observations (including friendly group, enemy group, and object group) and extract different high-dimensional state representations of different groups. CDN is designed based on a variational auto-encoder to allow each agent to choose its neighbors (communication targets) adaptively with its environment cognition difference. SCN is designed to handle the complex interactions among the agents with a soft attention mechanism. The results of simulations demonstrate the superior effectiveness of our method compared with existing methods.