Scalable Reinforcement Learning Research Articles

Efficient and scalable reinforcement learning for large-scale network control

The primary challenge in the development of large-scale artificial intelligence (AI) systems lies in achieving scalable decision-making—extending the AI models while maintaining sufficient performance. Existing research indicates that distributed AI can improve scalability by decomposing complex tasks and distributing them across collaborative nodes. However, previous technologies suffered from compromised real-world applicability and scalability due to the massive requirement of communication and sampled data. Here we develop a model-based decentralized policy optimization framework, which can be efficiently deployed in multi-agent systems. By leveraging local observation through the agent-level topological decoupling of global dynamics, we prove that this decentralized mechanism achieves accurate estimations of global information. Importantly, we further introduce model learning to reinforce the optimal policy for monotonic improvement with a limited amount of sampled data. Empirical results on diverse scenarios show the superior scalability of our approach, particularly in real-world systems with hundreds of agents, thereby paving the way for scaling up AI systems.

Efficient multi-agent cooperation: Scalable reinforcement learning with heterogeneous graph networks and limited communication

This paper addresses the challenge of scalable multi-agent reinforcement learning (MARL) under partial observability and communication constraints. An Efficient Multi-Agent Cooperation (EMAC) approach is proposed, which comprises a Heterogeneous Graph Network (HGN) and a Frame-wise Communication Reduction Algorithm (FCRA). HGN can effectively extract semantic information from local observations, while FCRA utilizes historical embeddings to achieve an approximation of multi-hop communication with 1-hop communication. Through comparative experiments, EMAC showed a 30% improvement in efficiency over the best-performing benchmark, scalability to arbitrary numbers of agents, and adaptation to unstructured environments. A t-SNE analysis of agent embeddings further reveals EMAC’s capability to understand and leverage semantic similarities among agents. These results highlight EMAC’s potential as a robust framework for decentralized multi-agent decision-making in complex and dynamic environments.

Fair and Scalable Orchestration of Network and Compute Resources for Virtual Edge Services

The combination of service virtualization and edge computing allows for low latency services, while keeping data storage and processing local. However, given the limited resources available at the edge, a conflict in resource usage arises when both virtualized user applications and network functions need to be supported. Further, the concurrent resource request by user applications and network functions is often entangled, since the data generated by the former has to be transferred by the latter, and vice versa. In this paper, we first show through experimental tests the correlation between a video-based application and a vRAN. Then, owing to the complex involved dynamics, we develop a scalable reinforcement learning framework for resource orchestration at the edge, which leverages a Pareto analysis for provable fair and efficient decisions. We validate our framework, named VERA, through a real-time proof-of-concept implementation, which we also use to obtain datasets reporting real-world operational conditions and performance. Using such experimental datasets, we demonstrate that VERA meets the KPI targets for over <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$96\%$</tex-math></inline-formula> of the observation period and performs similarly when executed in our real-time implementation, with KPI differences below 12.4%. Further, its scaling cost is <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$54\%$</tex-math></inline-formula> lower than a centralized framework based on deep-Q networks.

Energy management strategy for fuel cell electric vehicles based on scalable reinforcement learning in novel environment

To optimize the hydrogen consumption, lifespan of the fuel cell (FC), and durability of the lithium-ion battery in fuel cell electric vehicles (FCEVs), energy management strategies (EMSs) have been developed and implemented in the vehicle control units (VCUs) by engineers. Recently, with the rapid development of artificial intelligence, reinforcement learning (RL)-based EMS have shown great promise among various strategies. However, RL-based EMS tends to exhibit relatively poorer performance when faced with novel environments owing to the requirement for learning. This study proposes an enhanced EMS, Scalable Learning in Novel Environment (SLNE)-based EMS. This EMS performs well in unknown and dynamic environments by integrating a memory library (ML) composed of the Dirichlet process (DRP) clustering algorithm combined with the Chinese restaurant process (CRP) using the expectation-maximization (EM) algorithm for updates and the deep Q-network (DQN). Simulation and comparison are conducted under two novel driving cycles to evaluate the performance of the dynamic programming (DP)-based EMS, rule-based EMS, DQN-based EMS, and SLNE-based EMS. By comparing reward, power, and degradation, the results indicate that the proposed SLNE-based EMS demonstrates excellent convergence and adaptability when facing a novel and dynamic environment. Additionally, the proposed SLNE-based EMS achieves approximately 5% improvement in fuel economy, approximately 4.5% reduction in fuel cell degradation rate, and improved durability of the lithium-ion battery, compared with DQN-based EMS.

Scalable-MADDPG-Based Cooperative Target Invasion for a Multi-USV System.

This article concentrates on proposing a scalable deep reinforcement learning (DRL) method for a multiple unmanned surface vehicle (multi-USV) system to operate cooperative target invasion. The multi-USV system, which is made up of multiple invaders, needs to invade target areas in a specified time. A novel scalable reinforcement learning (RL) method called Scalable-MADDPG is proposed for the first time. In this method, the scale of the multi-USV system can be changed at any time without interrupting the training process. Then, to mitigate the policy oscillation after applying Scalable-MADDPG, a bi-directional long-short-term memory (Bi-LSTM) network is constructed. Moreover, an improved ϵ -greedy strategy is proposed to help balance the exploration and exploitation in RL. Furthermore, to enhance the robustness of the optimal policy, Ornstein-Uhlenbeck (OU) noise is added in this improved ϵ -greedy strategy during the training process. Finally, the scalable RL method is used to help the multi-USV system perform cooperative target invasion under complex marine environments. The effectiveness of Scalable-MADDPG is demonstrated through three experiments.

Scalable Reinforcement Learning Framework for Traffic Signal Control Under Communication Delays

Scalable Reinforcement Learning Framework for Traffic Signal Control Under Communication Delays

Scalable reinforcement learning approaches for dynamic pricing in ride-hailing systems

Dynamic pricing is a widely applied strategy by ride-hailing companies, such as Uber and Lyft, to match the trip demand with the availability of drivers. Deciding proper pricing policies is challenging and existing reinforcement learning (RL)-based solutions are restricted in solving small-scale problems. In this study, we contribute to RL-based approaches that can address the dynamic pricing problem in real-world-scale ride-hailing systems. We first characterize the dynamic pricing problem with a clear distinction between historical prices and current prices. We then translate our dynamic pricing problem into Markov Decision Process (MDP) and prove the existence of a deterministic stationary optimal policy. Our solutions are based on an off-policy reinforcement learning algorithm called twin-delayed deep determinant policy gradient (TD3) that performs offline learning of the optimal pricing policy using historical data and applies the learned policy to the next time slot, e.g., one week. We enhance TD3 by creating three mechanisms to reduce our model complexity and enhance training effectiveness. Extensive numerical experiments are conducted on both small grid networks (16 zones) and the NYC network (242 zones) to demonstrate the performance of the proposed algorithm. The results show our algorithm can efficiently find the optimal pricing policy for both the small and large networks, and can significantly enhance the platform profit and service efficiency.

Vehicle dynamic dispatching using curriculum-driven reinforcement learning

This study focuses on optimizing resource allocation problems in complex dynamic environments, specifically vehicle dispatching in closed bipartite queuing networks. We present a novel curriculum-driven reinforcement learning (RL) approach that seamlessly incorporates domain knowledge and environmental feedback, effectively addressing the challenges associated with sparse reward scenarios in RL applications. This approach involves a scalable reinforcement learning framework for dynamic vehicle fleet size. We design dense artificial rewards using domain knowledge and incorporate artificial action–reward pairs into the original experience sequence forming the basic structure of the training instances. A difficulty momentum boosting strategy is proposed to produce a series of training instances with progressively increasing difficulty, ensuring that the RL agent learns decision strategies in an organized and smooth manner. Experimental results demonstrate that the proposed method significantly surpasses existing approaches in enhancing productivity and model learning efficiency for transport tasks in open-pit mines, while confirming the superiority of a flexible and automated curriculum learning process over a rigid setting. This approach has vast potential for application in dynamic resource allocation problems across industries, such as manufacturing and logistics.

SCN_GNN: A GNN-based fraud detection algorithm combining strong node and graph topology information

Graph neural networks (GNNs) have exhibited remarkable success in fraud detection. However, detecting fraud in datasets with scattered graphic densities and multiple relations remains challenging. This is a prevalent issue in fraud detection as fraudsters often employ diverse relationship types to camouflage their activities. Moreover, the constrained interconnectivity between nodes contributes to a scarcity of informative data, thereby intensifying the influence of raw features and further compounding the difficulties in fraud detection processes. To address these challenges, we present SCN_GNN (Strongly Connected Nodes-Graph Neural Network), a novel algorithm for fraud detection, that proposes two node sampling strategies based on the fusion of strong node information and graph topology information. Among them, the structured similarity-aware module (SSAM) performs up-sampling to add useful nodes to the sparse graph, while the strong node module (SNM) performs down-sampling based on strong node information and original features. Furthermore, we also reconfigure the RSRL (Recursive and Scalable Reinforcement Learning framework) module to improve fraud detection performance by increasing inter-class distances and decreasing intra-class distances, resulting in a refined decision boundary for optimized algorithmic efficacy. We use three metrics (AUC, recall, and G_Mean) to evaluate the performance of SCN_GNN. The experimental results compared with the state-of-the-art models on two real-world datasets demonstrate the superiority of the proposed SCN_GNN.

A recursive reinforced blockchain performance evaluation and improvement architecture: Maximising diversity to improve scalability

SummaryNowadays, Blockchain technology has received widespread attention because of its ability to effectively solve the trust problem in transactions. However, the throughput of crypto currency using Blockchain technology has never been comparable to that of centralized payment institutions. The fundamental reason is the architecture of the Blockchain itself, and the existing methods often focus on the part of the Blockchain but lack the overall control. In this paper, we design, a novel Recursively Reinforced Blockchain Architecture Search (BORAS) architecture, from the perspective of the overall structure of the Blockchain, which can search the various attributes of the various levels in a adaptive manner, aiming at selecting the best configuration in a dynamical environment. First, we introduce a recursive and scalable reinforcement learning framework, to find the optimal block size and block interval of the Blockchain, so as to improve the throughput of the Blockchain system. Second, we add the selection scheme of consensus algorithm to the dynamic space and also use the framework to select the optimal consensus mechanism to realize the optimal configuration of the Blockchain system in the dynamic environment. Third, we adopt a recursive method, which greatly reduces the time complexity required for the optimal configuration of the Blockchain system and improves the configuration efficiency. Comprehensive experiments on three representative Blockchain platforms demonstrate that BOBAS can consistently improve the throughput, while also reducing latency, with remarkable and efficient performances.

A Survey of Scalable Reinforcement Learning

A Survey of Scalable Reinforcement Learning

Scalable Reinforcement Learning for Multiagent Networked Systems

Highlighted by success stories like AlphaGo, reinforcement learning (RL) has emerged as a powerful tool for decision making in complex environments. However, the success of RL has thus far been limited to small-scale or single-agent systems. To apply RL to large-scale networked systems such as energy, transportation, and communication networks, a critical hurdle is the curse of dimensionality, because for these systems, the state and action space can be exponentially large in the number of nodes in the network. This article attempts to break this curse of dimensionality and designs a scalable RL method, named scalable actor critic (SAC), for large networked systems. The key technical contribution is to exploit the network structure to derive an exponential decay property, which enables the design of the SAC approach.

Special Issue of INFORMS Journal on Computing—Scalable Reinforcement Learning Algorithms

Free AccessAboutSectionsView PDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissionsReprints ShareShare onFacebookTwitterLinked InEmail Go to SectionFree Access HomeINFORMS Journal on ComputingVol. 34, No. 1 Special Issue of INFORMS Journal on Computing—Scalable Reinforcement Learning AlgorithmsJ. Paul Brooks, Ted Ralphs, Nicola SecomandiJ. Paul Brooks, Ted Ralphs, Nicola SecomandiPublished Online:27 Jan 2022https://doi.org/10.1287/ijoc.2021.1153"Special Issue of INFORMS Journal on Computing—Scalable Reinforcement Learning Algorithms." INFORMS Journal on Computing, 34(1), pp. 2–3 Previous Back to Top Next FiguresReferencesRelatedInformation Volume 34, Issue 1January-February 2022Pages 1-669, C2 Article Information Metrics Downloaded 339 times in the past 12 months Information Published Online:January 27, 2022 Copyright © 2022, INFORMSCite asJ. Paul Brooks, Ted Ralphs, Nicola Secomandi (2022) Special Issue of INFORMS Journal on Computing—Scalable Reinforcement Learning Algorithms. INFORMS Journal on Computing 34(1):2-3. https://doi.org/10.1287/ijoc.2021.1153 PDF download

Reinforced Neighborhood Selection Guided Multi-Relational Graph Neural Networks

Graph Neural Networks (GNNs) have been widely used for the representation learning of various structured graph data, typically through message passing among nodes by aggregating their neighborhood information via different operations. While promising, most existing GNNs oversimplify the complexity and diversity of the edges in the graph and thus are inefficient to cope with ubiquitous heterogeneous graphs, which are typically in the form of multi-relational graph representations. In this article, we propose RioGNN , a novel Reinforced, recursive, and flexible neighborhood selection guided multi-relational Graph Neural Network architecture, to navigate complexity of neural network structures whilst maintaining relation-dependent representations. We first construct a multi-relational graph, according to the practical task, to reflect the heterogeneity of nodes, edges, attributes, and labels. To avoid the embedding over-assimilation among different types of nodes, we employ a label-aware neural similarity measure to ascertain the most similar neighbors based on node attributes. A reinforced relation-aware neighbor selection mechanism is developed to choose the most similar neighbors of a targeting node within a relation before aggregating all neighborhood information from different relations to obtain the eventual node embedding. Particularly, to improve the efficiency of neighbor selecting, we propose a new recursive and scalable reinforcement learning framework with estimable depth and width for different scales of multi-relational graphs. RioGNN can learn more discriminative node embedding with enhanced explainability due to the recognition of individual importance of each relation via the filtering threshold mechanism. Comprehensive experiments on real-world graph data and practical tasks demonstrate the advancements of effectiveness, efficiency, and the model explainability, as opposed to other comparative GNN models.

Scalable Deep Reinforcement Learning for Routing and Spectrum Access in Physical Layer

This paper proposes a novel scalable reinforcement learning approach for simultaneous routing and spectrum access in wireless ad-hoc networks. In most previous works on reinforcement learning for network optimization, the network topology is assumed to be fixed, and a different agent is trained for each transmission node—this limits scalability and generalizability. Further, routing and spectrum access are typically treated as separate tasks. Moreover, the optimization objective is usually a cumulative metric along the route, e.g., number of hops or delay. In this paper, we account for the physical-layer signal-to-interference-plus-noise ratio (SINR) in a wireless network and further show that <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">bottleneck</i> objective such as the minimum SINR along the route can also be optimized effectively using reinforcement learning. Specifically, we propose a scalable approach in which a single agent is associated with each flow and makes routing and spectrum access decisions as it moves along the frontier nodes. The agent is trained according to the physical-layer characteristics of the environment using a novel rewarding scheme based on the Monte Carlo estimation of the future bottleneck SINR. It learns to avoid interference by intelligently making joint routing and spectrum allocation decisions based on the geographical location information of the neighbouring nodes.

Mean-Field Multi-Agent Reinforcement Learning: A Decentralized Network Approach

One of the challenges for multi-agent reinforcement learning (MARL) is designing efficient learning algorithms for a large system in which each agent has only limited or partial information of the entire system. In this system, it is desirable to learn policies of a decentralized type. A recent and promising paradigm to analyze such decentralized MARL is to take network structures into consideration. While exciting progress has been made to analyze decentralized MARL with the network of agents, often found in social networks and team video games, little is known theoretically for decentralized MARL with the network of states, frequently used for modeling self-driving vehicles, ride-sharing, and data and traffic routing. This paper proposes a framework called localized training and decentralized execution to study MARL with network of states, with homogeneous (a.k.a. mean-field type) agents. Localized training means that agents only need to collect local information in their neighboring states during the training phase; decentralized execution implies that, after the training stage, agents can execute the learned decentralized policies, which only requires knowledge of the agents’ current states. The key idea is to utilize the homogeneity of agents and regroup them according to their states, thus the formulation of a networked Markov decision process with teams of agents, enabling the update of the Q-function in a localized fashion. In order to design an efficient and scalable reinforcement learning algorithm under such a framework, we adopt the actor-critic approach with over-parameterized neural networks, and establish the convergence and sample complexity for our algorithm, shown to be scalable with respect to the size of both agents and states.

Gnu-RL: A Practical and Scalable Reinforcement Learning Solution for Building HVAC Control Using a Differentiable MPC Policy

We propose Gnu-RL: a novel approach that enables real-world deployment of reinforcement learning (RL) for building control and requires no prior information other than historical data from existing Heating Ventilation and Air Conditioning (HVAC) controllers. In contrast with existing RL agents, which are opaque to expert interrogation and need ample training to achieve reasonable control performance, Gnu-RL is much more attractive for real-world applications. Furthermore, Gnu-RL avoids the need to develop and calibrate simulation environments for each building or system under control, thus making it highly scalable. To achieve this, we bootstrap the Gnu-RL agent with domain knowledge and expert demonstration. Specifically, Gnu-RL adopts a recently-developed Differentiable Model Predictive Control (MPC) policy, which encodes domain knowledge on planning and system dynamics, making it both sample-efficient and interpretable. Prior to any interaction with the environment, a Gnu-RL agent is pre-trained on historical data using imitation learning, enabling it to match the behavior of the existing controller. Once it is put in charge of controlling the environment, the agent continues to improve its policy end-to-end, using a policy gradient algorithm. We evaluate Gnu-RL in both simulation studies and a real-world testbed. Firstly, we validated the Gnu-RL agent is indeed practical and scalable by applying it to three commercial reference buildings in simulation, and demonstrated the superiority of the Differentiable MPC policy compared to a generic neural network. In another simulation experiment, our approach saved 6.6% of the total energy compared to the best published RL result for the same environment, while maintaining a higher level of occupant comfort. Finally, Gnu-RL was deployed to control the airflow to a real-world conference room with a Variable-Air-Volume (VAV) system during a three-week period. Our results show that Gnu-RL reduced cooling demand by 16.7% compared to the existing controller and tracked the temperature set-point better.

Scalable reinforcement learning for plant-wide control of vinyl acetate monomer process

This paper explores a reinforcement learning (RL) approach that designs automatic control strategies in a large-scale chemical process control scenario as the first step for leveraging an RL method to intelligently control real-world chemical plants. The huge number of units for chemical reactions as well as feeding and recycling the materials of a typical chemical process induces a vast amount of samples and subsequent prohibitive computation complexity in RL for deriving a suitable control policy due to high-dimensional state and action spaces. To tackle this problem, a novel RL algorithm: Factorial Fast-food Dynamic Policy Programming (FFDPP) is proposed. By introducing a factorial framework that efficiently factorizes the action space, Fast-food kernel approximation that alleviates the curse of dimensionality caused by the high dimensionality of state space, into Dynamic Policy Programming (DPP) that achieves stable learning even with insufficient samples. FFDPP is evaluated in a commercial chemical plant simulator for a Vinyl Acetate Monomer (VAM) process. Experimental results demonstrate that without any knowledge of the model, the proposed method successfully learned a stable policy with reasonable computation resources to produce a larger amount of VAM product with comparative performance to a state-of-the-art model-based control.

Scalable reinforcement learning on Cray XC

SummaryRecent advancements in deep learning have made reinforcement learning (RL) applicable to a much broader range of decision making problems. However, the emergence of reinforcement learning workloads brings multiple challenges to system resource management. RL applications continuously train a deep learning or a machine learning model while interacting with uncertain simulation models. This new generation of AI applications impose significant demands on system resources such as memory, storage, network, and compute. In this paper, we describe a typical RL application workflow and introduce the Ray distributed execution framework developed at the UC Berkeley RISELab. Ray includes the RLlib library for executing distributed reinforcement learning applications. We describe a recipe for deploying the Ray execution framework on Cray XC systems and demonstrate scaling of RLlib algorithms across multiple nodes of the system. We also explore performance characteristics across multiple CPU and GPU node types.

A Scalable Reinforcement Learning Algorithm for Scheduling Railway Lines

This paper describes an algorithm for scheduling bidirectional railway lines (both single- and multi-track) using a reinforcement learning (RL) approach. The goal is to define the track allocations and arrival/departure times for all trains on the line, given their initial positions, priority, halt times, and traversal times, while minimizing the total priority-weighted delay. The primary advantage of the proposed algorithm compared to exact approaches is its scalability, and compared to heuristic approaches is its solution quality. Efficient scaling is ensured by decoupling the size of the state-action space from the size of the problem instance. Improved solution quality is obtained because of the inherent adaptability of reinforcement learning to specific problem instances. An additional advantage is that the learning from one instance can be transferred with minimal re-learning to another instance with different infrastructure resources and traffic mix. It is shown that the solution quality of the RL algorithm exceeds that of two prior heuristic-based approaches while having comparable computation times. Two lines from the Indian rail network are used for demonstrating the applicability of the proposed algorithm in the real world.