Multi-task Deep Reinforcement Learning Research Articles

Not All Tasks Are Equally Difficult: Multi-Task Deep Reinforcement Learning with Dynamic Depth Routing

Multi-task reinforcement learning endeavors to accomplish a set of different tasks with a single policy. To enhance data efficiency by sharing parameters across multiple tasks, a common practice segments the network into distinct modules and trains a routing network to recombine these modules into task-specific policies. However, existing routing approaches employ a fixed number of modules for all tasks, neglecting that tasks with varying difficulties commonly require varying amounts of knowledge. This work presents a Dynamic Depth Routing (D2R) framework, which learns strategic skipping of certain intermediate modules, thereby flexibly choosing different numbers of modules for each task. Under this framework, we further introduce a ResRouting method to address the issue of disparate routing paths between behavior and target policies during off-policy training. In addition, we design an automatic route-balancing mechanism to encourage continued routing exploration for unmastered tasks without disturbing the routing of mastered ones. We conduct extensive experiments on various robotics manipulation tasks in the Meta-World benchmark, where D2R achieves state-of-the-art performance with significantly improved learning efficiency.

Joint Optimization of Power Generation and Voyage Scheduling in Ship Power System Based on Operating Scene Clustering and Multi-Task Deep Reinforcement Learning

Joint Optimization of Power Generation and Voyage Scheduling in Ship Power System Based on Operating Scene Clustering and Multi-Task Deep Reinforcement Learning

Multi-Task Deep Reinforcement Learning for Terahertz NOMA Resource Allocation With Hybrid Discrete and Continuous Actions

Multi-Task Deep Reinforcement Learning for Terahertz NOMA Resource Allocation With Hybrid Discrete and Continuous Actions

Multitask Transfer Deep Reinforcement Learning for Timely Data Collection in Rechargeable-UAV-Aided IoT Networks

Multitask Transfer Deep Reinforcement Learning for Timely Data Collection in Rechargeable-UAV-Aided IoT Networks

PiCor: Multi-Task Deep Reinforcement Learning with Policy Correction

Multi-task deep reinforcement learning (DRL) ambitiously aims to train a general agent that masters multiple tasks simultaneously. However, varying learning speeds of different tasks compounding with negative gradients interference makes policy learning inefficient. In this work, we propose PiCor, an efficient multi-task DRL framework that splits learning into policy optimization and policy correction phases. The policy optimization phase improves the policy by any DRL algothrim on the sampled single task without considering other tasks. The policy correction phase first constructs an adaptive adjusted performance constraint set. Then the intermediate policy learned by the first phase is constrained to the set, which controls the negative interference and balances the learning speeds across tasks. Empirically, we demonstrate that PiCor outperforms previous methods and significantly improves sample efficiency on simulated robotic manipulation and continuous control tasks. We additionally show that adaptive weight adjusting can further improve data efficiency and performance.

Multi-Task Reinforcement Learning for Distribution System Voltage Control With Topology Changes

This letter proposes a multi-task deep reinforcement learning (DRL) approach for distribution system voltage regulation considering topology changes via PV smart inverter control. The key idea is to encode the topology as an additional state for the DRL and leverage the multi-task learning scheme for joint learning of all task control policies. Unlike other DRL-based methods, our approach is robust to different topologies. Comparison results on the modified IEEE 123-node system demonstrate the enhanced robustness of the proposed method.

Intelligent Joint Network Slicing and Routing via GCN-Powered Multi-Task Deep Reinforcement Learning

In 6G mobile systems, network slicing is an emerging technology to support services with distinct requirements by dividing a common infrastructure into multiple logical networks. However, as a network management method, it is difficult for network slicing to achieve a real-time resource allocation to satisfy the stringent requirement of services in 6G networks. This paper introduces a joint network slicing and routing mechanism, which combines the network management and control framework to provide a fine-grained, real-time and dynamic resource allocation. Graph Convolutional Networks (GCN)-powered Multi-Task Deep Reinforcement Learning (DRL) is proposed to solve this complicated resource allocation problem. We first extend the DRL model into multi-task manner, where multiple output branches are matched to joint scheduling resources in every network slice. GCN with differentiable pooling mechanism is integrated into DRL model to capture the topological information from graph-structured network status. We implement our model in the SDN controller and evaluate it with representative topologies. The packet-level experiments show that 1) compared to rule-based and other learning-based methods, GCN-powered multi-task DRL can improve the performance of joint network slicing and routing; 2) our method is robust to diverse network environments; 3) in contrast with other learning-based algorithms, our method achieves a better performance.

Joint bidding and pricing for electricity retailers based on multi-task deep reinforcement learning

The single-task deep reinforcement learning (STDRL)-based methods solve the joint bidding and pricing problem for the electricity retailer in a hierarchical electricity market by defining a bidding policy and a pricing policy separately, which may suffer from low learning efficiency, time-consuming training and local optimization. To deal with these issues, this paper proposes a novel Multi-task Deep reinforcement learning approach for Joint Bidding and Pricing (MDJBP) optimization model. MDJBP can deal with the bidding and pricing tasks concurrently through a shared long short-term memory (LSTM) representation network to distill meaningful temporal characteristics from high-dimensional environment states. Furthermore, we develop a deep neural network (DNN) structure consisting a regression branch for bidding task and a soft actor-critic (SAC) branch for pricing task with automating entropy adjustment and adaptive loss weighting to implement MDJBP. The proposed multi-task deep reinforcement learning (MTDRL)-based method is tested with the IEEE 30-bus system. Numerical results show that the proposed methodology succeeds in giving an optimal joint bidding and pricing policy by fully exploiting commonalities and differences between bidding task and pricing task, and thereby boosts the profit, improves learning efficiency, reduces training time, and enhances stability.

Fully distributed actor-critic architecture for multitask deep reinforcement learning

AbstractWe propose a fully distributed actor-critic architecture, named diffusion-distributed-actor-criticDiff-DAC, with application to multitask reinforcement learning (MRL). During the learning process, agents communicate their value and policy parameters to their neighbours, diffusing the information across a network of agents with no need for a central station. Each agent can only access data from its local task, but aims to learn a common policy that performs well for the whole set of tasks. The architecture is scalable, since the computational and communication cost per agent depends on the number of neighbours rather than the overall number of agents. We derive Diff-DAC from duality theory and provide novel insights into the actor-critic framework, showing that it is actually an instance of the dual-ascent method. We prove almost sure convergence of Diff-DAC to a common policy under general assumptions that hold even for deep neural network approximations. For more restrictive assumptions, we also prove that this common policy is a stationary point of an approximation of the original problem. Numerical results on multitask extensions of common continuous control benchmarks demonstrate that Diff-DAC stabilises learning and has a regularising effect that induces higher performance and better generalisation properties than previous architectures.

Multi-task deep reinforcement learning for intelligent multi-zone residential HVAC control

In this short communication, a data-driven deep reinforcement learning (deep RL) method is applied to minimize HVAC users’ energy consumption costs while maintaining users’ comfort. The applied deep RL method's efficiency is enhanced by conducting multi-task learning that can achieve an economic control strategy for a multi-zone residential HVAC system in both cooling and heating scenarios. The applied multi-task deep RL method is compared with a rule-based benchmark case and a single-task deep deterministic policy gradient algorithm to verify its effective and generalized application in optimizing HVAC operation.

Scalable Parallel Task Scheduling for Autonomous Driving Using Multi-Task Deep Reinforcement Learning

The Internet of Vehicles (IoV) as a promising application of Internet of Things (IoT) has played a significant role in autonomous driving, by connecting intelligent vehicles. Autonomous driving needs to process the mass environmental sensing data in coordination with surrounding vehicles, and makes an accurate driving judgment accordingly. Since the vehicles always have limited computing resources, processing these data in parallel with efficient task scheduling is one of the most important topics. Most current work focuses on formulating special scenarios and service requirements as optimization problems. However, the complicated and dynamic environment of vehicular computing is hard to model, predict and control, making those previous methods unscalable and unable to reflect the real scenario. In this paper, a Multi-task Deep reinforcement learning approach for scalable parallel Task Scheduling (MDTS) is firstly devised. For avoiding the curse of dimensionality when coping with complex parallel computing environments and jobs with diverse properties, we extend the action selection in Deep Reinforcement Learning (DRL) to a multi-task decision, where the output branches of multi- task learning are fine-matched to parallel scheduling tasks. Child tasks of a job are accordingly assigned to distributed nodes without any human knowledge while the resource competition among parallel tasks is leveraged through shared neural network layers. Moreover, we design an appropriate reward function to optimize multiple metrics simultaneously, with emphasis on specific scenarios. Extensive experiments show that the MDTS significantly increases the overall reward compared with least- connection scheduling and particle swarm optimization algorithm from −16.71, −0.67 to 2.93, respectively.

A Survey of Multi-Task Deep Reinforcement Learning

Driven by the recent technological advancements within the field of artificial intelligence research, deep learning has emerged as a promising representation learning technique across all of the machine learning classes, especially within the reinforcement learning arena. This new direction has given rise to the evolution of a new technological domain named deep reinforcement learning, which combines the representational learning power of deep learning with existing reinforcement learning methods. Undoubtedly, the inception of deep reinforcement learning has played a vital role in optimizing the performance of reinforcement learning-based intelligent agents with model-free based approaches. Although these methods could improve the performance of agents to a greater extent, they were mainly limited to systems that adopted reinforcement learning algorithms focused on learning a single task. At the same moment, the aforementioned approach was found to be relatively data-inefficient, particularly when reinforcement learning agents needed to interact with more complex and rich data environments. This is primarily due to the limited applicability of deep reinforcement learning algorithms to many scenarios across related tasks from the same environment. The objective of this paper is to survey the research challenges associated with multi-tasking within the deep reinforcement arena and present the state-of-the-art approaches by comparing and contrasting recent solutions, namely DISTRAL (DIStill & TRAnsfer Learning), IMPALA(Importance Weighted Actor-Learner Architecture) and PopArt that aim to address core challenges such as scalability, distraction dilemma, partial observability, catastrophic forgetting and negative knowledge transfer.

IPAS: A deep Monte Carlo Tree Search-based intelligent pilot-power allocation scheme for massive MIMO system

Massive Multiple-Input-Multiple-Output (MIMO) is a promising technology to meet the demand for the connection of massive devices and high data capacity for mobile networks in the next generation communication system. However, due to the massive connectivity of mobile devices, the pilot contamination problem will severely degrade the communication quality and spectrum efficiency of the massive MIMO system. We propose a deep Monte Carlo Tree Search (MCTS)-based intelligent Pilot-power Allocation Scheme (iPAS) to address this issue. The core of iPAS is a multi-task deep reinforcement learning algorithm that can automatically learn the radio environment and make decisions on the pilot sequence and power allocation to maximize the spectrum efficiency with self-play training. To accelerate the searching convergence, we introduce a Deep Neural Network (DNN) to predict the pilot sequence and power allocation actions. The DNN is trained in a self-supervised learning manner, where the training data is generated from the searching process of the MCTS algorithm. Numerical results show that our proposed iPAS achieves a better Cumulative Distribution Function (CDF) of the ergodic spectral efficiency compared with the previous suboptimal algorithms.

Congestion Control in SDN-Based Networks via Multi-Task Deep Reinforcement Learning

Congestion control is a fundamental network task that modulates the data transmission rates of traffic sources to efficiently utilize network capacity. With the advent of machine learning, congestion control based on deep reinforcement learning is the subject of extensive attention. At present, research on machine-learning-based congestion control is mainly focused on single-task scenarios -- to design a control strategy that is only used to reduce congestion. This article studies the congestion control problem based on machine learning in multi-task scenarios. Specifically, we propose a congestion control model based on multi-task deep reinforcement learning. The model takes congestion control as the main task and load balancing as the auxiliary task. Compared to the single-task method, our model can better represent the network environment by learning the shared representation of congestion features and load balancing features. Moreover, network traffic control may involve both congestion control and load balancing. Therefore, learning multiple tasks jointly while exploiting commonalities and differences across tasks can help reduce the cost of task coordination. We use software defined networking to decouple the data and control planes, making network control strategies more flexible than traditional networks. To the best of our knowledge, this is the first time multi-task learning has been applied to network traffic control. Experimental results show that the method is efficient.

A Meta-Learning Framework for Learning Multi-User Preferences in QoE Optimization of DASH

Dynamic adaptive video streaming over hypertext transfer protocol (DASH) plays a key role in video transmission over the Internet. The conventional DASH adaptation approaches mainly focus on optimizing the overall quality of experience (QoE) for all client sides, neglecting the QoE diversity of different users. In this paper, we propose a meta-learning framework for multi-user preferences (MLMP) as a new DASH adaptation approach, which is able to optimize the diverse QoE of different users. Specifically, we first design a subjective experiment to analyze the difference of QoE preferences across users, in which QoE refers to the metrics of visual quality, fluctuation, and rebuffering events. Based on our findings, we formulate the QoE optimization of multi-user preferences as a multi-task deep reinforcement learning (DRL) problem. In our formulation, the QoE preference of each user is modeled in the overall QoE calculation via assigning the weights to the three QoE metrics. Then, the MLMP framework is developed to solve the proposed multi-task DRL problem, such that the preferences regarding visual quality, fluctuation, and rebuffering events can be optimized for different users in DASH adaptation. Finally, the simulation results show that the proposed approach outperforms state-of-the-art DASH adaptation approaches in satisfying the different users' QoE preferences regarding visual quality, fluctuation, and rebuffering events.

IRAF: A Deep Reinforcement Learning Approach for Collaborative Mobile Edge Computing IoT Networks

Recently, as the development of artificial intelligence (AI), data-driven AI methods have shown amazing performance in solving complex problems to support the Internet of Things (IoT) world with massive resource-consuming and delay-sensitive services. In this paper, we propose an intelligent resource allocation framework (iRAF) to solve the complex resource allocation problem for the collaborative mobile edge computing (CoMEC) network. The core of iRAF is a multitask deep reinforcement learning algorithm for making resource allocation decisions based on network states and task characteristics, such as the computing capability of edge servers and devices, communication channel quality, resource utilization, and latency requirement of the services, etc. The proposed iRAF can automatically learn the network environment and generate resource allocation decision to maximize the performance over latency and power consumption with self-play training. iRAF becomes its own teacher: a deep neural network (DNN) is trained to predict iRAF’s resource allocation action in a self-supervised learning manner, where the training data is generated from the searching process of Monte Carlo tree search (MCTS) algorithm. A major advantage of MCTS is that it will simulate trajectories into the future, starting from a root state, to obtain a best action by evaluating the reward value. Numerical results show that our proposed iRAF achieves 59.27% and 51.71% improvement on service latency performance compared with the greedy-search and the deep $Q$ -learning-based methods, respectively.

Multi-Task Deep Reinforcement Learning with PopArt

The reinforcement learning (RL) community has made great strides in designing algorithms capable of exceeding human performance on specific tasks. These algorithms are mostly trained one task at the time, each new task requiring to train a brand new agent instance. This means the learning algorithm is general, but each solution is not; each agent can only solve the one task it was trained on. In this work, we study the problem of learning to master not one but multiple sequentialdecision tasks at once. A general issue in multi-task learning is that a balance must be found between the needs of multiple tasks competing for the limited resources of a single learning system. Many learning algorithms can get distracted by certain tasks in the set of tasks to solve. Such tasks appear more salient to the learning process, for instance because of the density or magnitude of the in-task rewards. This causes the algorithm to focus on those salient tasks at the expense of generality. We propose to automatically adapt the contribution of each task to the agent’s updates, so that all tasks have a similar impact on the learning dynamics. This resulted in state of the art performance on learning to play all games in a set of 57 diverse Atari games. Excitingly, our method learned a single trained policy - with a single set of weights - that exceeds median human performance. To our knowledge, this was the first time a single agent surpassed human-level performance on this multi-task domain. The same approach also demonstrated state of the art performance on a set of 30 tasks in the 3D reinforcement learning platform DeepMind Lab.

Expert Level Control of Ramp Metering Based on Multi-Task Deep Reinforcement Learning

This paper shows how the recent breakthroughs in reinforcement learning (RL) that have enabled robots to learn to play arcade video games, walk, or assemble colored bricks, can be used to perform other tasks that are currently at the core of engineering cyberphysical systems. We present the first use of RL for the control of systems modeled by discretized non-linear partial differential equations (PDEs) and devise a novel algorithm to use non-parametric control techniques for large multi-agent systems. Cyberphysical systems (e.g., hydraulic channels, transportation systems, the energy grid, and electromagnetic systems) are commonly modeled by PDEs, which historically have been a reliable way to enable engineering applications in these domains. However, it is known that the control of these PDE models is notoriously difficult. We show how neural network-based RL enables the control of discretized PDEs whose parameters are unknown, random, and time-varying. We introduce an algorithm of mutual weight regularization (MWR), which alleviates the curse of dimensionality of multi-agent control schemes by sharing experience between agents while giving each agent the opportunity to specialize its action policy so as to tailor it to the local parameters of the part of the system it is located in. A discretized PDE, such as the scalar Lighthill–Whitham–Richards PDE can indeed be considered as a macroscopic freeway traffic simulator and which presents the most salient challenges for learning to control large cyberphysical system with multiple agents. We consider two different discretization procedures and show the opportunities offered by applying deep reinforcement for continuous control on both. Using a neural RL PDE controller on a traffic flow simulation based on a Godunov discretization of the San Francisco Bay Bridge, we are able to achieve precise adaptive metering without model calibration thereby beating the state of the art in traffic metering. Furthermore, with the more accurate BeATS simulator, we manage to achieve a control performance on par with ALINEA, a state-of-the-art parametric control scheme, and show how using MWR improves the learning procedure.