Multi-agent Setting Research Articles

Worst-case Bounds on Power vs. Proportion in Weighted Voting Games with an Application to False-name Manipulation

Weighted voting games apply to a wide variety of multi-agent settings. They enable the formalization of power indices which quantify the coalitional power of players. We take a novel approach to the study of the power of big vs. small players in these games. We model small (big) players as having single (multiple) votes. The aggregate relative power of big players is measured w.r.t. their votes proportion. For this ratio, we show small constant worst-case bounds for the Shapley-Shubik and the Deegan-Packel indices. In sharp contrast, this ratio is unbounded for the Banzhaf index. As an application, we define a false-name strategic normal form game where each big player may split its votes between false identities, and study its various properties. Together, our results provide foundations for the implications of players’ size, modeled as their ability to split, on their relative power.

LSTM-Based Channel Access Scheme for Vehicles in Cognitive Vehicular Networks With Multi-Agent Settings

In this paper, we study the channel access problem of vehicles in a cognitive radio vehicular network, where each vehicle opportunistically accesses the channel resources of the primary network in order to successfully receive the necessary data packets within a time deadline. Given the access priority constraint and the limited bandwidth of the primary network, a smart channel connection scheme is indispensable to ensure a decent quality of service (QoS) at the vehicles’ side. Due to the competitive nature of vehicles, the vehicle access control is formulated as a multi-agent access problem that comes with an intrinsic challenge, i.e. the partial observation of the information about the environment dynamics. On top of that, considering the temporal usage profile of the primary network, the environment dynamics are also time-dependant, and hence making the aforementioned access control a non-Markovian problem. Consequently, the estimation of the system states, which are used for the decision making process of a vehicle, is very challenging. To deal with the issues arising from such non-Markovian problem, we propose a vehicle connection algorithm based on a deep recurrent Q-learning network. With the aid of a recurrent Long Short Term Memory (LSTM) layer integrated into a deep Q-network, the time-correlated system states can be properly estimated, thereby improving the vehicle channel access policy. Besides, we introduce novel reward quantities that help improving the network performance and the capability to flexibly adapt to unexplored scenarios. A new structure of the cumulative reward function is also presented to balance the performance trade off between the cooperative and competitive objectives. Simulation results are provided to verify the advantage and the stability of our proposed algorithm over the benchmark schemes.

Evaluating Strategic Structures in Multi-Agent Inverse Reinforcement Learning

A core question in multi-agent systems is understanding the motivations for an agent's actions based on their behavior. Inverse reinforcement learning provides a framework for extracting utility functions from observed agent behavior, casting the problem as finding domain parameters which induce such a behavior from rational decision makers. We show how to efficiently and scalably extend inverse reinforcement learning to multi-agent settings, by reducing the multi-agent problem to N single-agent problems while still satisfying rationality conditions such as strong rationality. However, we observe that rewards learned naively tend to lack insightful structure, which causes them to produce undesirable behavior when optimized in games with different players from those encountered during training. We further investigate conditions under which rewards or utility functions can be precisely identified, on problem domains such as normal-form and Markov games, as well as auctions, where we show we can learn reward functions that properly generalize to new settings.

An improved approximation algorithm for maximin shares

Fair division is a fundamental problem in various multi-agent settings, where the goal is to divide a set of resources among agents in a fair manner. We study the case where m indivisible items need to be divided among n agents with additive valuations using the popular fairness notion of maximin share (MMS). An MMS allocation provides each agent a bundle worth at least her maximin share. While it is known that such an allocation need not exist [1,2], a series of remarkable work [1,3–6] provided approximation algorithms for a 23-MMS allocation in which each agent receives a bundle worth at least 23 times her maximin share. More recently, Ghodsi et al. [7] showed the existence of a 34-MMS allocation and a PTAS to find a (34−ϵ)-MMS allocation for an ϵ>0. Most of the previous works utilize intricate algorithms and require agents' approximate MMS values, which are computationally expensive to obtain.In this paper, we develop a new approach that gives a simple algorithm for showing the existence of a 34-MMS allocation. Furthermore, our approach is powerful enough to be easily extended in two directions: First, we get a strongly polynomial time algorithm to find a 34-MMS allocation, where we do not need to approximate the MMS values at all. Second, we show that there always exists a (34+112n)-MMS allocation, improving the best previous factor. This improves the approximation guarantee, most notably for small n. We note that 34 was the best factor known for n>4.

Analysing factorizations of action-value networks for cooperative multi-agent reinforcement learning

Recent years have seen the application of deep reinforcement learning techniques to cooperative multi-agent systems, with great empirical success. However, given the lack of theoretical insight, it remains unclear what the employed neural networks are learning, or how we should enhance their learning power to address the problems on which they fail. In this work, we empirically investigate the learning power of various network architectures on a series of one-shot games. Despite their simplicity, these games capture many of the crucial problems that arise in the multi-agent setting, such as an exponential number of joint actions or the lack of an explicit coordination mechanism. Our results extend those in Castellini et al. (Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS’19.International Foundation for Autonomous Agents and Multiagent Systems, pp 1862–1864, 2019) and quantify how well various approaches can represent the requisite value functions, and help us identify the reasons that can impede good performance, like sparsity of the values or too tight coordination requirements.

Subgradient averaging for multi-agent optimisation with different constraint sets

We consider a multi-agent setting with agents exchanging information over a possibly time-varying network, aiming at minimising a separable objective function subject to constraints. To achieve this objective we propose a novel subgradient averaging algorithm that allows for non-differentiable objective functions and different constraint sets per agent. Allowing different constraints per agent simultaneously with a time-varying communication network constitutes a distinctive feature of our approach, extending existing results on distributed subgradient methods. To highlight the necessity of dealing with a different constraint set within a distributed optimisation context, we analyse a problem instance where an existing algorithm does not exhibit a convergent behaviour if adapted to account for different constraint sets. For our proposed iterative scheme we show asymptotic convergence of the iterates to a minimum of the underlying optimisation problem for step sizes of the form ηk+1, η>0. We also analyse this scheme under a step size choice of ηk+1, η>0, and establish a convergence rate of O(lnkk) in objective value. To demonstrate the efficacy of the proposed method, we investigate a robust regression problem and an ℓ2 regression problem with regularisation.

Entropy Based Independent Learning in Anonymous Multi-Agent Settings

Efficient sequential matching of supply and demand is a problem of interest in many online to offline services. For instance, Uber, Lyft, Grab for matching taxis to customers; Ubereats, Deliveroo, FoodPanda etc for matching restaurants to customers. In these online to offline service problems, individuals who are responsible for supply (e.g., taxi drivers, delivery bikes or delivery van drivers) earn more by being at the ”right” place at the ”right” time. We are interested in developing approaches that learn to guide individuals to be in the ”right” place at the ”right” time (to maximize revenue) in the presence of other similar ”learning” individuals and only local aggregated observation of other agents states (e.g., only number of other taxis in same zone as current agent).

Towards Realizing Intelligent Coordinated Controllers for Multi-USV Systems Using Abstract Training Environments

Unmanned autonomous vehicles for various civilian and military applications have become a particularly interesting research area. Despite their many potential applications, a related technological challenge is realizing realistic coordinated autonomous control and decision making in complex and multi-agent environments. Machine learning approaches have been largely employed in simplified simulations to acquire intelligent control systems in multi-agent settings. However, the complexity of the physical environment, unrealistic assumptions, and lack of abstract physical environments derail the process of transition from simulation to real systems. This work presents a modular framework for automated data acquisition, training, and the evaluation of multiple unmanned surface vehicles controllers that facilitate prior knowledge integration and human-guided learning in a closed-loop. To realize this, we first present a digital maritime environment of multiple unmanned surface vehicles that abstracts the real-world dynamics in our application domain. Then, a behavior-driven artificial immune-inspired fuzzy classifier systems approach that is capable of optimizing agents’ behaviors and action selection in a multi-agent environment is presented. Evaluation scenarios of different combat missions are presented to demonstrate the performance of the system. Simulation results show that the resulting controllers can achieved an average wining rate between 52% and 98% in all test cases, indicating the effectiveness of the proposed approach and its feasibility in realizing adaptive controllers for efficient multiple unmanned systems’ cooperative decision making. We believe that this system can facilitate the simulation, data acquisition, training, and evaluation of practical cooperative unmanned vehicles’ controllers in a closed-loop.

Multi-featured collective perception with Evidence Theory: tackling spatial correlations

Collective perception allows sparsely distributed agents to form a global view on a common spatially distributed problem without any direct access to global knowledge and only based on a combination of locally perceived information. However, the evidence gathered from the environment is often subject to spatial correlations and depends on the movements of the agents. The latter is not always easy to control and the main question is how to share and to combine the estimated information to achieve the most precise global estimate in the least possible time. The current article aims at answering this question with the help of evidence theory, also known as Dempster–Shafer theory, applied to the collective perception scenario as a collective decision-making problem. We study eight most common belief combination operators to address the arising conflict between different sources of evidence in a highly dynamic multi-agent setting, driven by modulation of positive feedback. In comparison with existing approaches, such as voter models, the presented framework operates on quantitative belief assignments of the agents based on the observation time of the options according to the agents’ opinions. The evaluated results on an extended benchmark set for multiple options (n>2) indicate that the proportional conflict redistribution (PCR) principle allows a collective of small size (N=20), occupying 3.5% of the surface, to successfully resolve the conflict between clustered areas of features and reach a consensus with almost 100% certainty up to n=5.

On the Approximation of Nash Equilibria in Sparse Win-Lose Multi-player Games

A polymatrix game is a multi-player game over n players, where each player chooses a pure strategy from a list of its own pure strategies. The utility of each player is a sum of payoffs it gains from the two player's game from all its neighbors, under its chosen strategy and that of its neighbor. As a natural extension to two-player games (a.k.a. bimatrix games), polymatrix games are widely used for multi-agent games in real world scenarios. In this paper we show that the problem of approximating a Nash equilibrium in a polymatrix game within the polynomial precision is PPAD-hard, even in sparse and win-lose ones. This result further challenges the predictability of Nash equilibria as a solution concept in the multi-agent setting. We also propose a simple and efficient algorithm, when the game is further restricted. Together, we establish a new dichotomy theorem for this class of games. It is also of independent interest for exploring the computational and structural properties in Nash equilibria.

Scalable Equilibrium Computation in Multi-agent Influence Games on Networks

We provide a polynomial-time, scalable algorithm for equilibrium computation in multi-agent influence games on networks, extending work of Bindel, Kleinberg, and Oren (2015) from the single-agent to the multi-agent setting. In games of influence, agents have limited advertising budget to influence the initial predisposition of nodes in some network towards their products, but the eventual decisions of the nodes are determined by the stationary state of DeGroot opinion dynamics on the network, which takes over after the seeding (Ahmadinejad et al. 2014, 2015). In multi-agent systems, how should agents spend their budgets to seed the network to maximize their utility in anticipation of other advertising agents and the network dynamics? We show that Nash equilibria of this game are pure and (under weak assumptions) unique, and can be computed in polynomial time; we test our model by computing equilibria using mirror descent for the two-agent case on random graphs.

Standing out from the crowd: Both cue numerosity and social information affect attention in multi-agent contexts.

Groups of people offer abundant opportunities for social interactions. We used a two-phase task to investigate how social cue numerosity and social information about an individual affected attentional allocation in such multi-agent settings. The learning phase was a standard gaze-cuing procedure in which a stimulus face could be either uninformative or informative about the upcoming target. The test phase was a group-cuing procedure in which the stimulus faces from the learning phase were presented in groups of three. The target could either be cued by the group minority (i.e., one face) or majority (i.e., two faces) or by uninformative or informative stimulus faces. Results showed an effect of cue numerosity, whereby responses were faster to targets cued by the group majority than the group minority. However, responses to targets cued by informative identities included in the group minority were as fast as responses to targets cued by the group majority. Thus, previously learned social information about an individual was able to offset the general enhancement of cue numerosity, revealing a complex interplay between cue numerosity and social information in guiding attention in multi-agent settings.

A Scalable Approach to Predict Multi-Agent Motion for Human-Robot Collaboration

Human motion prediction is considered a key component for enabling fluent human-robot collaboration. The ability to anticipate the motion and subsequent intent of the partner(s) remains a challenging task due to the complex and interpersonal nature of human behavior. In this work, we propose a novel sequence learning approach that learns a robust representation over the observed human motion and can condition future predictions over a subset of past sequences. Our approach works for both single and multi-agent settings and relies on an interpretable latent space that has the implicit benefit of improving human motion understanding. We evaluated the proposed approach by comparing its performance against state-of-the-art motion prediction methods on single, multi-agent, and human-robot collaboration datasets. The results suggest that our approach outperforms other methods over all the evaluated temporal horizons, for single-agent and multi-agent motion prediction. The improved performance of our approach for both single and multi-agent settings, coupled with an interpretable latent space, can enable close-proximity human-robot collaboration.

A projective simulation scheme for partially observable multi-agent systems

We introduce classical partial observability to the projective simulation (PS) learning method by adding a belief projection operator and an observability parameter to the original framework of the PS efficiency. The formulation is expanded on the Dirac notation for further quantum applications. We show that both game-theoretical notions of cooperation and competition are assignable to the PS multi-agent setting via, e.g., classes of coalitions, and Nash equilibriums, respectively. In any case, network representations and situated scenarios are suggested from the invasion toy problem.

Distributed resource allocation with binary decisions via Newton-like Neural Network dynamics

This paper aims to solve a distributed resource allocation problem with binary local constraints. The problem is formulated as a binary program with a cost function defined by the summation of agent costs plus a global mismatch/penalty term. We propose a modification of the Hopfield Neural Network (HNN) dynamics in order to solve this problem while incorporating a novel Newton-like weighting factor. This addition lends itself to fast avoidance of saddle points, which the gradient-like HNN is susceptible to. Turning to a multi-agent setting, we reformulate the problem and develop a distributed implementation of the Newton-like dynamics. We show that if a local solution to the distributed reformulation is obtained, it is also a local solution to the centralized problem. A main contribution of this work is to show that the probability of converging to a saddle point of an appropriately defined energy function in both the centralized and distributed settings is zero under light assumptions. Finally, we enlarge our algorithm with an annealing technique which gradually learns a feasible binary solution. Simulation results demonstrate that the proposed methods are competitive with centralized greedy and SDP relaxation approaches in terms of solution quality, while the main advantage of our approach is a significant improvement in runtime over the SDP relaxation method and the distributed quality of implementation.

IHG-MA: Inductive heterogeneous graph multi-agent reinforcement learning for multi-intersection traffic signal control

Multi-agent deep reinforcement learning (MDRL) has been widely applied in multi-intersection traffic signal control. The MDRL algorithms produce the decentralized cooperative traffic-signal policies via specialized multi-agent settings in certain traffic networks. However, the state-of-the-art MDRL algorithms seem to have some drawbacks. (1) It is desirable that the traffic-signal policies can be smoothly transferred to diverse traffic networks, however, the adopted specialized multi-agent settings hinder the traffic-signal policies to transfer and generalize to new traffic networks. (2) Existing MDRL algorithms which are based on deep neural networks cannot flexibly tackle a time-varying number of vehicles traversing the traffic networks. (3) Existing MDRL algorithms which are based on homogeneous graph neural networks fail to capture the heterogeneous features of objects in traffic networks. Motivated by the above observations, in this paper, we propose an algorithm, referred to as Inductive Heterogeneous Graph Multi-agent Actor–critic (IHG-MA) algorithm, for multi-intersection traffic signal control. The proposed IHG-MA algorithm has two features: (1) It conducts representation learning using a proposed inductive heterogeneous graph neural network (IHG), which is an inductive algorithm. The proposed IHG algorithm can generate embeddings for previously unseen nodes (e.g., new entry vehicles) and new graphs (e.g., new traffic networks). But unlike the algorithms based on the homogeneous graph neural network, IHG algorithm not only encodes heterogeneous features of each node, but also encodes heterogeneous structural (graph) information. (2) It also conducts policy learning using a proposed multi-agent actor–critic(MA), which is a decentralized cooperative framework. The proposed MA framework employs the final embeddings to compute the Q-value and policy, and then optimizes the whole algorithm via the Q-value and policy loss. Experimental results on different traffic datasets illustrate that IHG-MA algorithm outperforms the state-of-the-art algorithms in terms of multiple traffic metrics, which seems to be a new promising algorithm for multi-intersection traffic signal control.

A Sufficient Statistic for Influence in Structured Multiagent Environments

Making decisions in complex environments is a key challenge in artificial intelligence (AI). Situations involving multiple decision makers are particularly complex, leading to computational intractability of principled solution methods. A body of work in AI has tried to mitigate this problem by trying to distill interaction to its essence: how does the policy of one agent influence another agent? If we can find more compact representations of such influence, this can help us deal with the complexity, for instance by searching the space of influences rather than the space of policies. However, so far these notions of influence have been restricted in their applicability to special cases of interaction. In this paper we formalize influence-based abstraction (IBA), which facilitates the elimination of latent state factors without any loss in value, for a very general class of problems described as factored partially observable stochastic games (fPOSGs). On the one hand, this generalizes existing descriptions of influence, and thus can serve as the foundation for improvements in scalability and other insights in decision making in complex multiagent settings. On the other hand, since the presence of other agents can be seen as a generalization of single agent settings, our formulation of IBA also provides a sufficient statistic for decision making under abstraction for a single agent. We also give a detailed discussion of the relations to such previous works, identifying new insights and interpretations of these approaches. In these ways, this paper deepens our understanding of abstraction in a wide range of sequential decision making settings, providing the basis for new approaches and algorithms for a large class of problems.

A logic of knowledge based on abstract arguments

AbstractWe introduce a logic of knowledge in a framework in which knowledge is treated as a kind of belief. The framework is based on a standard KD45 characterization of belief, and the characterization of knowledge undergoes the classical tripartite analysis that knowledge is justified true belief, which has a natural link to the studies of logics of evidence and justification. The interpretation of knowledge avoids the unwanted properties of logical omniscience, independent of the choice of the base logic of belief. We axiomatize the logic, prove its soundness and completeness and study the computational complexity results of the model checking and satisfiability problems. We extend the logic to a multi-agent setting and introduce a variant in which belief is characterized in a weaker system to avoid the problem of logical omniscience.

Communication-Aware Multi-Agent Metareasoning for Decentralized Task Allocation

<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Metareasoning</i> refers to reasoning about one’s own decision making process. This paper considers metareasoning about the decision making process in multi-agent settings. We present a multi-agent metareasoning approach that enables a multi-agent team to select which task allocation algorithm to use as a function of changing communication quality level. Given a set of multi-agent task allocation algorithms, we synthesize a policy that prescribes the best algorithm to use among a predefined set of algorithms for a given communication level. Since each agent in the team runs the same policy, the team (or a part of the team) will collectively switch between task allocation algorithms as a function of the observed level of communication. We apply reactive synthesis to generate the policy from high-level specifications written in Linear Temporal Logic encoding the agents’ switching behavior with respect to the state of the environment. We perform experiments in simulation to identify the best performing algorithms under different communication levels. The communication environment is modeled using the Rayleigh fading model and communication estimation is done through the exchange of heartbeat messages among agents. We test our metareasoning policy in three types of scenarios: search & rescue, fire monitoring, and ship protection scenarios. For each scenario, we demonstrate that our policy achieved better performance with respect to either max distance traveled, max number of transmitted messages or both compared to running any single algorithm.

Dynamic path finding method and obstacle avoidance for automated guided vehicle navigation in Industry 4.0

Within the scope of Industry 4.0, Automated Guided Vehicles (AGVs) are used to streamline logistics through the usage of efficient path finding methods. The current path finding methods in the industry rely on excessive usage of guidance in the shape of magnets, tapes or QR codes on the floor that the AGVs follow to reach their destinations. However, the current methods lack operational flexibility and are costly to scale in the cases of job-shop floor expansions. In this paper, a dynamic path finding method with obstacle avoidance is presented which utilizes distance measuring sensors to avoid obstacles and reach the goal destination using the most direct path as possible. Tests for functionality and multi-agent scaling have been conducted to evaluate the performance of the dynamic method in a multi-agent setting. The results show that the dynamic method scales properly and is capable of navigating multiple agents through a simulated warehouse environment autonomously and without relying on external guidance. The dynamic method is able to avoid most collisions using distance measuring sensors and multi-agent negotiation to resolve conflicts among the agents that could have resulted in potential collisions. The proposed dynamic method provides a flexible and scalable path finding method for use in Industry 4.0.