Multi-agent Learning Research Articles

Internet Traffic Prediction with Distributed Multi-Agent Learning

Internet traffic prediction has been considered a research topic and the basis for intelligent network management and planning, e.g., elastic network service provision and content delivery optimization. Various methods have been proposed in the literature for Internet traffic prediction, including statistical, machine learning and deep learning methods. However, most of the existing approaches are trained and deployed in a centralized approach, without considering the realistic scenario in which multiple parties are concerned about the prediction process and the prediction model can be trained in a distributed approach. In this study, a distributed multi-agent learning framework is proposed to fill the research gap and predict Internet traffic in a distributed approach, in which each agent trains a base prediction model and the individual models are further aggregated with the cooperative interaction process. In the numerical experiments, two sophisticated deep learning models are chosen as the base prediction model, namely, long short-term memory (LSTM) and gated recurrent unit (GRU). The numerical experiments demonstrate that the GRU model trained with five agents achieves state-of-the-art performance on a real-world Internet traffic dataset collected in a campus backbone network in terms of root mean square error (RMSE) and mean absolute error (MAE).

Entropy regularized actor-critic based multi-agent deep reinforcement learning for stochastic games

Multi-agent reinforcement learning (MARL) is an abstract framework modeling a dynamic environment that involves multiple learning and decision-making agents, each of which tries to maximize her cumulative reward. In MARL, each agent discovers a strategy alongside others and adapts her policy in response to the behavioural changes of others. A fundamental difficulty faced by MARL is that every agent is dynamically learning and changing to improve her reward, making the whole system unstable and agents’ policies difficult to converge. In this paper, we introduce the entropy regularizer into the Bellman equation and utilize Lagrange approach to optimize the entropy regularizer. We then propose a MARL algorithm based on the maximum entropy principle and the actor-critic method. This algorithm follows the policy gradient approach and uses a policy network and a value network. We call it Multi-Agent Deep Soft Policy Gradient (MADSPG). Then by using the Lagrange approach and dynamic minimax optimization, we propose the AUTO-MADSPG algorithm with an automatically adjusted entropy regularizer. These algorithms make multi-agent learning more stable while sufficient exploration is guaranteed. Finally, we also incorporate MADSPG with the recently proposed opponent modeling component into an integrated framework. This framework outperforms many state-of-the-art MARL algorithms in conventional cooperative and competitive game settings.

Many-to-Many Data Aggregation Scheduling Based on Multi-Agent Learning for Multi-Channel WSN

Many-to-many data aggregation has become an indispensable technique to realize the simultaneous executions of multiple applications with less data traffic load and less energy consumption in a multi-channel WSN (wireless sensor network). The problem of how to efficiently allocate time slot and channel for each node is one of the most critical problems for many-to-many data aggregation in multi-channel WSNs, and this problem can be solved with the new distributed scheduling method without communication conflict outlined in this paper. The many-to-many data aggregation scheduling process is abstracted as a decentralized partially observable Markov decision model in a multi-agent system. In the case of embedding cooperative multi-agent learning technology, sensor nodes with group observability work in a distributed manner. These nodes cooperated and exploit local feedback information to automatically learn the optimal scheduling strategy, then select the best time slot and channel for wireless communication. Simulation results show that the new scheduling method has advantages in performance when comparing with the existing methods.

Learning multi-agent cooperation.

Advances in reinforcement learning (RL) have resulted in recent breakthroughs in the application of artificial intelligence (AI) across many different domains. An emerging landscape of development environments is making powerful RL techniques more accessible for a growing community of researchers. However, most existing frameworks do not directly address the problem of learning in complex operating environments, such as dense urban settings or defense-related scenarios, that incorporate distributed, heterogeneous teams of agents. To help enable AI research for this important class of applications, we introduce the AI Arena: a scalable framework with flexible abstractions for associating agents with policies and policies with learning algorithms. Our results highlight the strengths of our approach, illustrate the importance of curriculum design, and measure the impact of multi-agent learning paradigms on the emergence of cooperation.

Designing all-pay auctions using deep learning and multi-agent simulation

We propose a multi-agent learning approach for designing crowdsourcing contests and All-Pay auctions. Prizes in contests incentivise contestants to expend effort on their entries, with different prize allocations resulting in different incentives and bidding behaviors. In contrast to auctions designed manually by economists, our method searches the possible design space using a simulation of the multi-agent learning process, and can thus handle settings where a game-theoretic equilibrium analysis is not tractable. Our method simulates agent learning in contests and evaluates the utility of the resulting outcome for the auctioneer. Given a large contest design space, we assess through simulation many possible contest designs within the space, and fit a neural network to predict outcomes for previously untested contest designs. Finally, we apply mirror ascent to optimize the design so as to achieve more desirable outcomes. Our empirical analysis shows our approach closely matches the optimal outcomes in settings where the equilibrium is known, and can produce high quality designs in settings where the equilibrium strategies are not solvable analytically.

CrowdIM: Crowd-Inspired Intelligent Manufacturing Space Design

Crowd-inspired intelligent manufacturing space (CrowdIM) aims to leverage the aggregated power of heterogeneous human–machine–things (HMT) agents for improving the efficiency of intelligent manufacturing. A significant scientific problem in CrowdIM is how to improve individual skills and crowd intelligence through cooperation, complementation, competition, and confrontation among HMT agents. The emergence mechanism of biological crowd intelligence provides an inspiration to address this challenge. This article explores the mapping mechanisms between natural crowd intelligence and CrowdIM, from the aspects, such as collective dynamics, self-adaptive mechanism, crowd intelligence optimization, graph structure mapping model, evolutionary game dynamics, multiagent learning, and so on. We further propose a general model of CrowdIM and expound it through a typical case study.

Decentralized Learning for Optimality in Stochastic Dynamic Teams and Games With Local Control and Global State Information

Stochastic dynamic teams and games are rich models for decentralized systems and challenging testing grounds for multiagent learning. Previous work that guaranteed team optimality assumed stateless dynamics, or an explicit coordination mechanism, or joint-control sharing. In this article, we present an algorithm with guarantees of convergence to team optimal policies in teams and common interest games. The algorithm is a two-timescale method that uses a variant of Q-learning on the finer timescale to perform policy evaluation while exploring the policy space on the coarser timescale. Agents following this algorithm are “independent learners”: they use only local controls, local cost realizations, and global state information, without access to controls of other agents. The results presented here are the first, to the best of our knowledge, to give formal guarantees of convergence to team optimality using independent learners in stochastic dynamic teams and common interest games.

Behavior Reasoning for Opponent Agents in Multi-Agent Learning Systems

One important component of developing autonomous agents lies in the accurate prediction of their opponents’ behaviors when the agents interact with others in an uncertain environment. Most recent study focuses on first constructing predictive types (or models) of the opponents, considering their various properties of interest, and subsequently using these models to predict their behaviors accordingly. However, as the possible type space can be rather large, it is time-consuming, and sometimes even infeasible, to predict the actual behaviors of opponents with all candidate types. Thus, in this paper a tractable opponent behavior reasoning approach is proposed that facilitates ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$a$</tex-math></inline-formula> ) extraction of a small yet representative summary of all candidates using sub-modular-type maximization, and accordingly, ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$b$</tex-math></inline-formula> ) identification of the most appropriate type for real-time behavior prediction based on multi-armed bandits. In addition, we propose a knowledge-transfer scheme through demonstration learning to synchronize subject agents’ knowledge about their opponents’ behaviors. This further reduces the burden of reasoning with all models of their opponents from the perspective of individual subject agents. We integrate the new behavior prediction and reasoning method into a state-of-the-art evolutionary multi-agent framework, namely a memetic multi-agent system (MeMAS), and demonstrate empirical performance in two problem domains.

Developing, evaluating and scaling learning agents in multi-agent environments

The Game Theory &amp; Multi-Agent team at DeepMind studies several aspects of multi-agent learning ranging from computing approximations to fundamental concepts in game theory to simulating social dilemmas in rich spatial environments and training 3-d humanoids in difficult team coordination tasks. A signature aim of our group is to use the resources and expertise made available to us at DeepMind in deep reinforcement learning to explore multi-agent systems in complex environments and use these benchmarks to advance our understanding. Here, we summarise the recent work of our team and present a taxonomy that we feel highlights many important open challenges in multi-agent research.

Traffic flow control using multi-agent reinforcement learning

One of the technologies based on information technology used today is the VANET network used for inter-road communication. Today, many developed countries use this technology to optimize travel times, queue lengths, number of vehicle stops, and overall traffic network efficiency. In this research, we investigate the critical and necessary factors to increase the quality of VANET networks. This paper focuses on increasing the quality of service using multi-agent learning methods. The innovation of this study is using artificial intelligence to improve the network’s quality of service, which uses a mechanism and algorithm to find the optimal behavior of agents in the VANET. The result indicates that the proposed method is more optimal in the evaluation criteria of packet delivery ratio (PDR), transaction success rate, phase duration, and queue length than the previous ones. According to the evaluation criteria, TSR 6.342%, PDR 9.105%, QL 7.143%, and PD 6.783% are more efficient than previous works.

Packet Routing in Dynamic Multi-Hop UAV Relay Network: A Multi-Agent Learning Approach

The multi-hop unmanned aerial vehicle (UAV) network can serve as data relays where ground users (GUs) do not have reliable direct connections to the base station (BS). Existing works mainly focus on simple dual-hop system. In this paper, we investigate the packet routing problem in a multi-hop UAV relay network to minimize the data transmission time and enhance the network throughput. However, the dynamic network topology due to UAV mobility makes the packet routing challenging since the limited communication range of each UAV leads to volatile wireless connection. Moreover, the line-of-sight communication links may cause strong interference among UAVs. Towards this end, we propose a novel multi-agent deep reinforcement learning based algorithm, named as multi-agent QMIX (MAQMIX) to: 1) design proper UAVs’ trajectories to serve the moving GUs while maintaining the network connection; 2) allocate frequency resource properly among UAVs to alleviate the impact of interference; and 3) choose a proper next hop UAV for each data packet to reduce the transmission time and probability of network congestion. The proposed MAQMIX has two novel training mechanisms, i.e., intra-UAV and inter-UAV training mechanisms, which can tackle the large action space issue and coordinate the training among UAVs in the multi-hop UAV relay network. Simulation results demonstrate that the MAQMIX outperforms baseline schemes in terms of the network congestion avoidance, throughput, and transmission time.

Multi-Agent Learning for the Inverse Kinematics of a Robotic Arm

This paper presents a solution to the inverse kinematics problem for robotic manipulator based on the Adaptive Multi-Agent System (AMAS) approach. In this research, multi-agent system is in charge of controlling a robot arm with four degrees of freedom (DOF) and two motorized wheels, giving appropriate commands, such as rotation angles and velocities, to reach the desired position and orientation of the end effector. The calculation of commands is directly related to the solving of forward and inverse kinematics. Before the learning process of AMOEBA, the rotational angles, θ values, are encoded into a single number N, this parameter is the desired value that we are going to predict in the predicting stage. During the learning phase, the Agnostic MOdEl Builder by self-Adaptation (AMOEBA) builds context agents, which has local models and is able to self-adapt. After the getting the predicted value, Npred, it will be decoded back to get the set of rotational angles that is given to robot end effector. In addition, the robot with all its physical parameters is modeled and simulated in the Robot Operating System (ROS) environment

Risk Adversarial Learning System for Connected and Autonomous Vehicle Charging

In this article, the design of a rational decision support system (RDSS) for a connected and autonomous vehicle charging infrastructure (CAV-CI) is studied. In the considered CAV-CI, the distribution system operator (DSO) deploys electric vehicle supply equipment (EVSE) to provide an electrical vehicle (EV) charging facility for human-driven connected vehicles (CVs) and AVs. The charging request by the human-driven EV becomes irrational when it demands more energy and charging period than its actual need. Therefore, the scheduling policy of each EVSE must be adaptively accumulated the irrational charging request to satisfy the charging demand of both CVs and autonomous vehicles (AVs). To tackle this, we formulate an RDSS problem for the DSO, where the objective is to maximize the charging capacity utilization by satisfying the laxity risk of the DSO. Thus, we devise a rational reward maximization problem to adapt the irrational behavior by CVs in a data-informed manner. We propose a novel risk adversarial multiagent learning system (RAMALS) for CAV-CI to solve the formulated RDSS problem. In RAMALS, the DSO acts as a centralized risk adversarial agent (RAA) for informing the laxity risk to each EVSE. Subsequently, each EVSE plays the role of a self-learner agent to adaptively schedule its own EV sessions by coping advice from RAA. The experiment results show that the proposed RAMALS affords around 46.6% improvement in charging rate, about 28.6% improvement in the EVSE’s active charging time, and at least 33.3% more energy utilization, as compared to a currently deployed ACN EVSE system, and other baselines.

Multi-Agent Reinforcement Learning for a Random Access Game

This work investigates a random access (RA) game for a time-slotted RA system, where <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$N$</tex-math></inline-formula> players choose a set of slots of a frame and each frame consists of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$M$</tex-math></inline-formula> multiple time slots. We obtain the pure strategy Nash equilibria (PNEs) of this RA game, where slots are fully utilized as in the centralized scheduling. As an algorithm to realize a PNE (Pure strategy Nash Equilibrium), we propose an Exponential-weight algorithm for Exploration and Exploitation (EXP3)-based multi-agent (MA) learning algorithm, which has the computational complexity of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$O(N N_{\max }^{2} T)$</tex-math></inline-formula> . EXP3 is a bandit algorithm designed to find an optimal strategy in a multi-armed bandit (MAB) problem that users do not know the expected payoff of each strategy. Our simulation results show that the proposed algorithm can achieve PNEs. Moreover, it can adapt to time-varying environments, where the number of players varies over time.

Efficient Application of Competitive Multiagent Learning by Neuroevolution

Multiagent frameworks give an ideal climate to the assessment and examination of true issues utilizing support learning calculations. Most customary ways to deal with multiagent learning are impacted by lengthy preparation periods as well as high computational intricacy. Flawless (NeuroEvolution of Augmenting Topologies) is a well-known developmental methodology used to get the best performing brain network design frequently used to handle streamlining issues in the field of man-made reasoning. This paper uses the NEAT calculation to accomplish serious multiagent learning on a changed pong game climate in a productive way. The contending specialists keep various guidelines while having comparable perception space boundaries. The proposed calculation uses this property of the climate to characterize a particular neuro transformative system that gets the ideal strategy for every one of the specialists. The ordered outcomes show that the proposed execution accomplishes ideal conduct in an exceptionally short preparation period when contrasted with existing multiagent support learning models.

Multiagent Self-Redundancy Identification and Tuned Greedy-Exploration.

The constant development of sensing applications using innovative and affordable measurement devices has increased the amount of data transmitted through networks, carrying in many cases, redundant information that requires more time to be analyzed or larger storage centers. This redundancy is mainly present because the network nodes do not recognize environmental variations requiring exploration, which causes a repetitive data collection in a set of limited locations. In this work, we propose a multiagent learning framework that uses the Gaussian process regression (GPR) to allow the agents to predict the environmental behavior by means of the neighborhood measurements, and the rate distortion function to establish a border in which the environmental information is neither misunderstood nor redundant. We apply this framework to a mobile sensor network and demonstrate that the nodes can tune the parameter s of the Blahut-Arimoto algorithm in order to adjust the gathered environment information and to become more or less exploratory within a sensing area.

An Information Fusion Approach to Intelligent Traffic Signal Control Using the Joint Methods of Multiagent Reinforcement Learning and Artificial Intelligence of Things

With the development of communication technology and artificial intelligence of things (AIoT), transportation systems have become much smarter than ever before. However, the volume of vehicles and traffic flows have rapidly increased. Optimizing and improving urban traffic signal control is a potential way to relieve traffic congestion. In general, traffic signal control is a sequential decision process that conforms to the characteristics of reinforcement learning, in which an agent constantly interacts with its environment, thus providing strategy for optimizing behavior in accordance with feedback in response. In this paper, we propose multiagent reinforcement learning for traffic signals (MARL4TS) to support the control and deployment of traffic signals. First, information on traffic flows and multiple intersections is formalized as input environments for performing reinforcement learning. Second, we design a new reward function to continuously select the most appropriate strategy as control during multiagent learning to track actions for traffic signals. Finally, we use a supporting tool, Simulation of Urban MObility (SUMO), to simulate the proposed traffic signal control process and compare it with other methods. The experimental results show that our proposed MARL4TS method is superior to the baselines. In particular, our method can reduce vehicle delay.

Cooperative Multi-Agent Fairness and Equivariant Policies

We study fairness through the lens of cooperative multi-agent learning. Our work is motivated by empirical evidence that naive maximization of team reward yields unfair outcomes for individual team members. To address fairness in multi-agent contexts, we introduce team fairness, a group-based fairness measure for multi-agent learning. We then prove that it is possible to enforce team fairness during policy optimization by transforming the team's joint policy into an equivariant map. We refer to our multi-agent learning strategy as Fairness through Equivariance (Fair-E) and demonstrate its effectiveness empirically. We then introduce Fairness through Equivariance Regularization (Fair-ER) as a soft-constraint version of Fair-E and show that it reaches higher levels of utility than Fair-E and fairer outcomes than non-equivariant policies. Finally, we present novel findings regarding the fairness-utility trade-off in multi-agent settings; showing that the magnitude of the trade-off is dependent on agent skill.

Equilibrium Learning in Auction Markets

My dissertation investigates the computation of Bayes-Nash equilibria in auctions via multiagent learning. A particular focus lies on the game-theoretic analysis of learned gradient dynamics in such markets. This requires overcoming several technical challenges like non-differentiable utility functions and infinite-dimensional strategy spaces. Positive results may open the door for wide-ranging applications in Market Design and the economic sciences.

Learning-Based Online QoE Optimization in Multi-Agent Video Streaming

Video streaming has become a major usage scenario for the Internet. The growing popularity of new applications, such as 4K and 360-degree videos, mandates that network resources must be carefully apportioned among different users in order to achieve the optimal Quality of Experience (QoE) and fairness objectives. This results in a challenging online optimization problem, as networks grow increasingly complex and the relevant QoE objectives are often nonlinear functions. Recently, data-driven approaches, deep Reinforcement Learning (RL) in particular, have been successfully applied to network optimization problems by modeling them as Markov decision processes. However, existing RL algorithms involving multiple agents fail to address nonlinear objective functions on different agents’ rewards. To this end, we leverage MAPG-finite, a policy gradient algorithm designed for multi-agent learning problems with nonlinear objectives. It allows us to optimize bandwidth distributions among multiple agents and to maximize QoE and fairness objectives on video streaming rewards. Implementing the proposed algorithm, we compare the MAPG-finite strategy with a number of baselines, including static, adaptive, and single-agent learning policies. The numerical results show that MAPG-finite significantly outperforms the baseline strategies with respect to different objective functions and in various settings, including both constant and adaptive bitrate videos. Specifically, our MAPG-finite algorithm maximizes QoE by 15.27% and maximizes fairness by 22.47% compared to the standard SARSA algorithm for a 2000 KB/s link.