Multi-agent Reinforcement Learning Approach Research Articles

Integrating Knowledge Compilation with Reinforcement Learning for Routes

Sequential multiagent decision-making under partial observability and uncertainty poses several challenges. Although multiagent reinforcement learning (MARL) approaches have increased the scalability, addressing combinatorial domains is still challenging as random exploration by agents is unlikely to generate useful reward signals. We address cooperative multiagent pathfinding under uncertainty and partial observability where agents move from their respective sources to destinations while also satisfying constraints (e.g., visiting landmarks). Our main contributions include: (1) compiling domain knowledge such as underlying graph connectivity and domain constraints into propositional logic based decision diagrams, (2) developing modular techniques to integrate such knowledge with deep MARL algorithms, and (3) developing fast algorithms to query the compiled knowledge for accelerated episode simulation in RL. Empirically, our approach can tractably represent various types of domain constraints, and outperforms previous MARL approaches significantly both in terms of sample complexity and solution quality on a number of instances.

A Cooperative Spectrum Sensing With Multi-Agent Reinforcement Learning Approach in Cognitive Radio Networks

Cognitive radio networks (CRNs) can greatly improve the temporal and spatial spectrum utilization by identifying and exploring spectrum holes of the licensed primary users (PUs). However, since the occupation of primary channels changes dynamically, a swift and accurate spectrum sensing is crucial especially in the multi-channel multi-secondary users (SUs) environment, where the number of channels is much larger than that of SUs. To improve the sensing accuracy, a cooperative sensing algorithm is proposed in this letter, where multiple SUs can share their spectrum detection results for a more effective spectrum holes search. This letter further employs multi-agent deep deterministic policy gradient (MADDPG) algorithm with the feature of centralized training and decentralized execution to reduce the synchronization and communication overhead caused by the sensing cooperation of SUs. The numerical simulation demonstrates that with the combination of cooperative sensing and multi-agent reinforcement learning, the proposed algorithm can greatly enhance the sensing accuracy in comparison to other non-cooperative learning or centralized learning approaches.

Multi-agent reinforcement learning approach for hedging portfolio problem.

Developing a hedging strategy to reduce risk of losses for a given set of stocks in a portfolio is a difficult task due to cost of the hedge. In Vietnam stock market, cross-hedge is involved hedging a long position of a stock because there is no put option for the stock. In addition, only VN30 stock index futures contracts are traded on Hanoi Stock Exchange. Inspired by recently achievement of deep reinforcement learning, we explore feasibility to construct a hedging strategy automatically by leveraging cooperative multi-agent in reinforcement learning techniques without advanced domain knowledge. In this work, we use 10 popular stocks on Ho Chi Minh Stock Exchange, and VN30F1M (VN30 Index Futures contracts within one month settlement) to develop a stock market simulator (including transaction fee, tax, and settlement date of transactions) for reinforcement learning agent training. We use daily return as input data for training process. Results suggest that the agent can learn trading and hedging policy to make profit and reduce losses. Furthermore, we also find that our agent can protect portfolios and make positive profit in case market collapses systematically. In practice, this work can help Vietnam’s stock market investors to improve performance and reduce losses in trading, especially when the volatility cannot be controlled.

Intelligent Anti-Jamming Communication for Wireless Sensor Networks: A Multi-Agent Reinforcement Learning Approach

In this article, we investigate intelligent anti-jamming communication method for wireless sensor networks. The stochastic game framework is introduced to model and analyze the multi-user anti-jamming problem, and a joint multi-agent anti-jamming algorithm (JMAA) is proposed to obtain the optimal anti-jamming strategy. In intelligent multi-channel blocking jamming environment, the proposed JMAA adopts multi-agent reinforcement learning to make online channel selection, which can effectively tackle the external malicious jamming and avoid the internal mutual interference among sensor nodes. The simulation results show that, the proposed JMAA is superior to the frequency-hopping method, the sensing-based method and the independent reinforcement learning. Specifically, the proposed JMAA has the higher average packet receive ratio than both the frequency-hopping method and the sensing-based method. Compared with the independent reinforcement learning, JMAA has faster convergence rate when reaching the same performance of average packet receive ratio. In addition, since the JMAA does not need to model the jamming patterns, it can be widely used for combating other malicious jamming such as sweep jamming and probabilistic jamming.

Multi-UAV Path Planning for Wireless Data Harvesting With Deep Reinforcement Learning

Harvesting data from distributed Internet of Things (IoT) devices with multiple autonomous unmanned aerial vehicles (UAVs) is a challenging problem requiring flexible path planning methods. We propose a multi-agent reinforcement learning (MARL) approach that, in contrast to previous work, can adapt to profound changes in the scenario parameters defining the data harvesting mission, such as the number of deployed UAVs, number, position and data amount of IoT devices, or the maximum flying time, without the need to perform expensive recomputations or relearn control policies. We formulate the path planning problem for a cooperative, non-communicating, and homogeneous team of UAVs tasked with maximizing collected data from distributed IoT sensor nodes subject to flying time and collision avoidance constraints. The path planning problem is translated into a decentralized partially observable Markov decision process (Dec-POMDP), which we solve through a deep reinforcement learning (DRL) approach, approximating the optimal UAV control policy without prior knowledge of the challenging wireless channel characteristics in dense urban environments. By exploiting a combination of centered global and local map representations of the environment that are fed into convolutional layers of the agents, we show that our proposed network architecture enables the agents to cooperate effectively by carefully dividing the data collection task among themselves, adapt to large complex environments and state spaces, and make movement decisions that balance data collection goals, flight-time efficiency, and navigation constraints. Finally, learning a control policy that generalizes over the scenario parameter space enables us to analyze the influence of individual parameters on collection performance and provide some intuition about system-level benefits.

Utilising Assured Multi-Agent Reinforcement Learning within Safety-Critical Scenarios

Multi-agent reinforcement learning allows a team of agents to learn how to work together to solve complex decision-making problems in a shared environment. However, this learning process utilises stochastic mechanisms, meaning that its use in safety-critical domains can be problematic. To overcome this issue, we propose an Assured Multi-Agent Reinforcement Learning (AMARL) approach that uses a model checking technique called quantitative verification to provide formal guarantees of agent compliance with safety, performance, and other non-functional requirements during and after the reinforcement learning process. We demonstrate the applicability of our AMARL approach in three different patrolling navigation domains in which multi-agent systems must learn to visit key areas by using different types of reinforcement learning algorithms (temporal difference learning, game theory, and direct policy search). Furthermore, we compare the effectiveness of these algorithms when used in combination with and without our approach. Our extensive experiments with both homogeneous and heterogeneous multi-agent systems of different sizes show that the use of AMARL leads to safety requirements being consistently satisfied and to better overall results than standard reinforcement learning.

Multi-Agent Reinforcement learning Approach to IoT Coordination

Nowadays, the IoT is evolving at a very fast pace and has proven its usefulness in several areas by creating better applications and services. However, more flexible approaches proving a well-defined architecture meeting the general requirements and building blocks of IoT are still needed despite the results obtained in the literature. In this paper, we focus on the IoT coordination challenge which represents a fundamental property allowing things to collaborate and make a decision when an appropriate change is detected in its environment. This contribution proposes an agent-based approach coupled with Q-learning which is a reinforcement learning technique, to compensate for coordination in its entirety, namely objective coordination and subjective coordination. To illustrate this approach, an evacuation use case is presented.

Automating turbulence modelling by multi-agent reinforcement learning

Turbulent flow models are critical for applications such as aircraft design, weather forecasting and climate prediction. Existing models are largely based on physical insight and engineering intuition. More recently, machine learning has been contributing to this endeavour with promising results. However, all efforts have focused on supervised learning, which is difficult to generalize beyond training data. Here we introduce multi-agent reinforcement learning as an automated discovery tool of turbulence models. We demonstrate the potential of this approach on large-eddy simulations of isotropic turbulence, using the recovery of statistical properties of direct numerical simulations as a reward. The closure model is a control policy enacted by cooperating agents, which detect critical spatio-temporal patterns in the flow field to estimate the unresolved subgrid-scale physics. Results obtained with multi-agent reinforcement learning algorithms based on experience replay compare favourably with established modelling approaches. Moreover, we show that the learned turbulence models generalize across grid sizes and flow conditions. Turbulence modelling is an essential flow simulation tool, but is typically dependent on physical insight and engineering intuition. Novati et al. develop a multi-agent reinforcement learning approach for learning turbulence models that can generalize across grid sizes and flow conditions.

A MARL Approach for Optimizing Positions of VANET Aerial Base-Stations on a Sparse Highway

A Vehicular Ad-Hoc Network (VANET) helps vehicles send and receive environmental and traffic information, making it a crucial component towards fully autonomous roads. For VANETs to serve their purpose, there has to be sufficient coverage, even in less populated areas. Moreover, a lot of the safety information is time-sensitive; excessive delay in data transfer can increase the risk of fatal accidents. Unmanned Aerial Vehicles (UAVs) can be used as mobile base-stations to fill in gaps of coverage. The placement of these UAVs is crucial towards how well the system performs. We are particularly interested in the placement of mobile base-stations for a rural highway with sparse traffic, as it represents the worst-case scenario for vehicular communication. Instead of heuristic or linear programming methods for optimal placement, we use multi-agent reinforcement learning (MARL). The main benefit of MARL is that it allows the agents to learn model-free through experience. We propose a variation of the traditional Deep Independent Q-Learning. The modifications include an observation function augmented with information directly shared between neighbouring agents as well a shared policy scheme. We also implement a custom sparse highway simulator that is used for training and testing our algorithm. Our testing shows that the proposed MARL algorithm is able to learn the placement policies that produce the maximum rewards for different scenarios while adapting to the dynamic road densities along the service area. Our experiments show that our model is scalable, allowing the number of agents to increase without any modifications to the code. Finally, we show that our model can be generalized as the algorithm can be directly used and performs equally as well on an industry standard simulator. Future experiments can be performed to improve the realism and complexity of the highway models and adapting the algorithm to real-world scenarios.

Distributed Vehicle Tracking in Wireless Sensor Network: A Fully Decentralized Multiagent Reinforcement Learning Approach

Vehicle tracking is one of the important applications of the wireless sensor network (WSN), and sensor scheduling is essential in WSN for achieving an efficient tracking process. Traditional centralized sensor scheduling frameworks cannot meet real-time requirements of vehicle tracking in WSN, because of task overloading caused by limited resource consumption and communication bandwidth. To solve this problem, this letter proposes a multiagent distributed sensor scheduling framework in WSN. This letter first proposes a preactivation-based vehicle tracking model to preactive some sensors in order to reduce unnecessary resource consumption. Then, this letter develops a fully decentralized multiagent reinforcement learning (FDMARL) algorithm to design our multiagent sensor scheduling framework. The simulation results show the convergence and the superiority of our proposed FDMARL-aided sensor scheduling algorithm.

Machine learning-assisted cross-slice radio resource optimization: Implementation framework and algorithmic solution

Network slicing is a central feature in 5G and beyond systems to allow operators to customize their networks for different applications and customers. With network slicing, different logical networks, i.e. network slices, with specific functional and performance requirements can be created over the same physical network. A key challenge associated with the exploitation of the network slicing feature is how to efficiently allocate underlying network resources, especially radio resources, to cope with the spatio-temporal traffic variability while ensuring that network slices can be provisioned and assured within the boundaries of Service Level Agreements / Service Level Specifications (SLAs/SLSs) with customers. In this field, the use of artificial intelligence, and, specifically, Machine Learning (ML) techniques, has arisen as a promising approach to cater for the complexity of resource allocation optimization among network slices. This paper tackles the description of a feasible implementation framework for deploying ML-assisted solutions for cross-slice radio resource optimization that builds upon the work conducted by 3GPP and O-RAN Alliance. On this basis, the paper also describes and evaluates an ML-assisted solution that uses a Multi-Agent Reinforcement Learning (MARL) approach based on the Deep Q-Network (DQN) technique and fits within the presented implementation framework.

Cooperative Caching and Fetching in D2D Communications - A Fully Decentralized Multi-Agent Reinforcement Learning Approach

To satisfy the increasing demands of cellular traffic, cooperative content caching at the network edge (e.g., User Equipment) has become a promising paradigm in the next-generation cellular networks. Device-to-Device (D2D) communications can improve the content caching and fetching performance without deploying additional infrastructure. In this paper, we investigate the joint optimization of cooperative caching and fetching in dynamic D2D environment for minimizing the overall content fetching delay. We formulate it as a decentralized partially observable Markov game for finding the optimal policies at agents. To address this problem, we propose a Fully Decentralized Soft Multi-Agent Reinforcement Learning (FDS-MARL) algorithm, which extends the soft actor-critic framework to non-stationary multi-agent environment for fully decentralized learning and it contains the following major design components: Graph Attention Network based self-attention for cooperative inter-agent coordination, a consensus communication mechanism for effectively reducing the information loss and non-stationarity of the environment while keeping gradual global consensus, and an influence based transmission scheduling mechanism for effective credit assignment and also alleviation of potential transmission contentions among agents. Simulation results show that FDS-MARL can improve the content caching and fetching performance significantly compared with the representative work in the literature.

Multi-vehicle routing problems with soft time windows: A multi-agent reinforcement learning approach

Multi-vehicle routing problem with soft time windows (MVRPSTW) is an indispensable constituent in urban logistics distribution systems. Over the past decade, numerous methods for MVRPSTW have been proposed, but most are based on heuristic rules that require a large amount of computation time. With the current rapid increase of logistics demands, traditional methods incur the dilemma between computational efficiency and solution quality. To efficiently solve the problem, we propose a novel reinforcement learning algorithm called the Multi-Agent Attention Model that can solve routing problem instantly benefit from lengthy offline training. Specifically, the vehicle routing problem is regarded as a vehicle tour generation process, and an encoder-decoder framework with attention layers is proposed to generate tours of multiple vehicles iteratively. Furthermore, a multi-agent reinforcement learning method with an unsupervised auxiliary network is developed for the model training. By evaluated on four synthetic networks with different scales, the results demonstrate that the proposed method consistently outperforms Google OR-Tools and traditional methods with little computation time. In addition, we validate the robustness of the well-trained model by varying the number of customers and the capacities of vehicles.

Edge Intelligence Empowered Urban Traffic Monitoring: A Network Tomography Perspective

Efficient urban traffic monitoring is a key enabler for intelligent planning and management of modern cities. Network tomography can monitor the urban traffic with a comparably small number of traffic detectors like cameras, and has become an appealing technique for urban traffic management. However, previous work on network tomography based traffic monitoring focuses primarily on developing estimators using the given end-to-end travel time measurements, while the design of data collection for efficiently distributed collecting and processing the raw monitoring videos to such measurements is often neglected. We fill this gap by exploring the vision of edge intelligence for optimal urban traffic monitoring, and tackle the following two problems in regard of limited telecommunications resources: 1) when the total number of monitoring videos that are successfully processed into the end-to-end travel time measurements is pre-bounded, we employ a Fisher Information Matrix (FIM) to help determine the best quota scheme for the monitoring videos that each traffic detector need to generate and 2) when the centralised processing of monitoring videos alone is insufficient, we make use of the computation capabilities from these edge devices, i.e., traffic detectors, and employ a multi-agent reinforcement learning approach to help them conduct intelligent computation offloading individually. Extensive simulations demonstrate that our proposed scheme effectively reduces the estimation error of network tomography compared to common approaches with either uniform or random strategy.

Enabling Sustainable Underwater IoT Networks With Energy Harvesting: A Decentralized Reinforcement Learning Approach

In this article, we study an energy sustainable Internet-of-Underwater Things (IoUT) network with tidal energy harvesting. Specifically, an analytical model is first developed to analyze the performance of the IoUT network, characterizing the stochastic nature of energy harvesting and traffic demands of IoUT nodes, and the salient features of acoustic communication channels. It is found that the spatial uncertainty resulting from underwater acoustic communication may cause a severe fairness issue. As such, an optimization problem is formulated to maximize the network throughput under fairness constraints, by tuning the random access parameters of each node. Given the global network information, including the number of nodes, energy harvesting rates, communication distances, etc., the optimization problem can be efficiently solved with the Branch and Bound (BnB) method. Considering a realistic network where the network information may not be available at the IoUT nodes, we further propose a multiagent reinforcement learning approach for each node to autonomously adapt the random access parameter based on the interactions with the dynamic network environment. The numerical results show that the proposed learning algorithm greatly improves the throughput performance compared with the existing solutions, and approaches the derived theoretical bound.

Autonomous Tracking Using a Swarm of UAVs: A Constrained Multi-Agent Reinforcement Learning Approach

In this paper, we aim to design an autonomous tracking system for a swarm of unmanned aerial vehicles (UAVs) to localize a radio frequency (RF) mobile target. In the system, UAVs equipped with omnidirectional received signal strength (RSS) sensors can cooperatively search the target with a specified tracking accuracy. To achieve fast localization and tracking in the highly dynamic channel environment (e.g., time-varying transmit power and intermittent signal), we formulate a flight decision problem as a constrained Markov decision process (CMDP) with the main objective of avoiding redundant UAV flight path. Then, we propose an enhanced multi-agent reinforcement learning to coordinate multiple UAVs performing real-time target tracking. The core of the proposed scheme is a feedback control system that takes into account the uncertainty of the channel estimate. We prove that the proposed algorithm can converge to the optimal decision. Our simulation results show that the proposed scheme outperforms standard Q-learning and multi-agent Q-learning algorithms in terms of searching time and successful localization probability.

Reward design for driver repositioning using multi-agent reinforcement learning

A large portion of passenger requests is reportedly unserviced, partially due to vacant for-hire drivers’ cruising behavior during the passenger seeking process. This paper aims to model the multi-driver repositioning task through a mean field multi-agent reinforcement learning (MARL) approach that captures competition among multiple agents. Because the direct application of MARL to the multi-driver system under a given reward mechanism will likely yield a suboptimal equilibrium due to the selfishness of drivers, this study proposes a reward design scheme with which a more desired equilibrium can be reached. To effectively solve the bilevel optimization problem with upper level as the reward design and the lower level as a multi-agent system, a Bayesian optimization (BO) algorithm is adopted to speed up the learning process. We then apply the bilevel optimization model to two case studies, namely, e-hailing driver repositioning under service charge and multiclass taxi driver repositioning under NYC congestion pricing. In the first case study, the model is validated by the agreement between the derived optimal control from BO and that from an analytical solution. With a simple piecewise linear service charge, the objective of the e-hailing platform can be increased by 8.4%. In the second case study, an optimal toll charge of $5.1 is solved using BO, which improves the objective of city planners by 7.9%, compared to that without any toll charge. Under this optimal toll charge, the number of taxis in the NYC central business district is decreased, indicating a better traffic condition, without substantially increasing the crowdedness of the subway system.

PSO-Based: MARL Approach for Frequency Regulation of Multi-area Power System

In this study, a particle swarm optimization (PSO) based Multi-Agent Reinforcement Learning (MARL) approach for load frequency control (LFC) of multi-area power system over communication network considering communication delay and packet loss is presented. A networked control system (NCS) is framed to model a constant delay and packet loss using Markovian approach. The controller using PSO based MARL technique having two levels of controlled action in each area of power system act as an estimator agent and controller agent respectively. The proposed method for frequency regulation of multi-area power system is studied in detail and verified for different types of load disturbances. In addition, to check the system dynamic performance of electric power system a mean square errors (MSEs) of states with different value of communication delay are computed. The simulation results of this study is validated and compared with some pre-design methods available from literatures, along with capacity of multi agent system (MAS) for application of LFC in power system over communication network.

Neighborhood Cognition Consistent Multi-Agent Reinforcement Learning

Social psychology and real experiences show that cognitive consistency plays an important role to keep human society in order: if people have a more consistent cognition about their environments, they are more likely to achieve better cooperation. Meanwhile, only cognitive consistency within a neighborhood matters because humans only interact directly with their neighbors. Inspired by these observations, we take the first step to introduce neighborhood cognitive consistency (NCC) into multi-agent reinforcement learning (MARL). Our NCC design is quite general and can be easily combined with existing MARL methods. As examples, we propose neighborhood cognition consistent deep Q-learning and Actor-Critic to facilitate large-scale multi-agent cooperations. Extensive experiments on several challenging tasks (i.e., packet routing, wifi configuration and Google football player control) justify the superior performance of our methods compared with state-of-the-art MARL approaches.

Multiagent Reinforcement Learning for Energy Management in Residential Buildings

The aim of this article is to explore the multiagent reinforcement learning approach for residential multicarrier energy management. Defining the multiagents system not only enhances the possibility of dedicating separate demand response programs for different components but also accelerates the computational calculations. We employ the Q -learning to provide the optimum solution in solving the presented residential energy management problem. Furthermore, to address uncertainties, a scenario-based method with the real data and proper probability density functions is used. Deterministic and stochastic numerical calculations are made to justify the effectiveness and robustness of the proposed method. The simulated results indicate that the application of the proposed reinforcement learning-based method leads to lower cost schemes for consumers rather than the conventional optimization-based energy management programs.