Multi-agent Decision Problems Research Articles

A logic Petri net model for dynamic multi-agent game decision-making

This study proposes a logical Petri net model to leverage the modeling advantages of Petri nets in handling batch processing and uncertainty in value passing and to integrate relevant game elements from multi-agent game processes for modeling multi-agent decision problems and resolving optimization issues in dynamic multi-agent game decision-making. Firstly, the attributes of each token are defined as rational agents, and utility function values and state probability transition functions are assigned to them. Secondly, decision transitions are introduced, and the triggering of the optimal decision transition is determined based on a comparison of token utility function values, along with an associated algorithm. Finally, a dynamic game emergency business decision-making process for sudden events is modeled and analyzed using the logic game decision Petri net. Based on reachable markings, reachable graphs are constructed to analyze the dynamic game process. Algorithms are described for the generation of reachable graphs, and the paper explores how the logic game decision model for sudden events can address dynamic game decision problems, generate optimal emergency plans, and analyze resource conflicts during emergency processes. The effectiveness and superiority of the model in analyzing the emergency business decision-making process for sudden events are validated.

Cache-Aided MEC for IoT: Resource Allocation Using Deep Graph Reinforcement Learning

With the growing demand for latency-sensitive and compute-intensive services in the Internet of Things (IoT), multi-access edge computing (MEC)-enabled IoT is envisioned as a promising technique that allows network nodes to have computing and caching capabilities. In this paper, we propose a cache-aided MEC (CA-MEC) offloading framework for joint optimization of communication, computing, and caching (3C) resources in the MEC-enabled IoT. Our goal is to optimize the offloading decision and resource allocation strategy to minimize the system latency subject to dynamic cache capacities and computing resource constraints. We first formulate this optimization problem as a multi-agent decision problem, a partially observable Markov decision process (POMDP). Then, the deep graph convolution reinforcement learning (DGRL) method is applied to motivate the agents to learn optimal strategies cooperatively in a highly dynamic environment. Simulations show that our method is highly effective for computation offloading and resource allocation and performs superior results in a large-scale network.

A Unified Approach to Dynamic Decision Problems With Asymmetric Information: Nonstrategic Agents

We study a general class of dynamic multi-agent decision problems with asymmetric information and non-strategic agents, which includes dynamic teams as a special case. When agents are non-strategic, an agent's strategy is known to the other agents. Nevertheless, the agents strategy choices and beliefs are interdependent over times, a phenomenon known as signaling. We introduce the notion of sufficient information that effectively compresses the agents information in a mutually consistent manner. Accordingly, we propose an information state for each agent that is sufficient for decision making purposes. We present instances where we can determine an information state with a time-invariant domain for each agent. We present a generalization of the policy-independence property of belief in Partially Observed Markov Decision Processes to dynamic multi-agent decision problems. Furthermore, we propose a sequential decomposition that decouples the interdependence between the agents strategies and beliefs over time, and enables us to formulate a dynamic program to determine a globally optimal policy in dynamic teams via backward induction.

Using of Rectangular Stochastic Matrices for the Problem of Evaluating and Ranking Alternatives

The paper investigates the issue related to a possible generalization of the “state-probability of choice” model so that the generalized model could be applied to the problem of ranking alternatives, either individual or by a group of agents. It is shown that the results obtained before for the problem of multi-agent choice and decision making by majority of votes can be easily transferred to the problem of multi-agent alternatives ranking. On the basis of distributions of importance values for the problem of ranking alternatives, we can move on to similar models for the choice and voting with the help of well-known exponential normalization of rows.So we regard two types of matrices, both of which belonging to the sort of matrices named balanced rectangular stochastic matrices. For such matrices, sums of elements in each row equal 1, and all columns have equal sums of elements. Both types are involved in a two-level procedure regarded in this paper. Firstly a matrix representing all possible distributions of importance among alternatives should be formed, and secondly a “state-probability of choice” matrix should be obtained on its base. For forming a matrix of states, which belongs and the rows of which correspond to possible distributions of importance, applying pairwise comparisons and the Analytic Hierarchy Method is suggested. Parameterized transitive scales with the parameter affecting the spread of importance between the best and the worst alternatives are regarded. For further getting the matrices of choice probabilities, another parameter which reflects the degree of the agent’s decisiveness is also introduced. The role of both parameters is discussed and illustrated with examples in the paper.The results are reported regarding some numerical experiments which illustrate getting distributions of importance on the basis of the Analytic Hierarchy Process and which are connected to gaining the situation of dynamic equilibrium of alternatives, i.e. the situation when alternatives are considered as those of equal value.

Toward data-driven solutions to interactive dynamic influence diagrams

With the availability of significant amount of data, data-driven decision making becomes an alternative way for solving complex multiagent decision problems. Instead of using domain knowledge to explicitly build decision models, the data-driven approach learns decisions (probably optimal ones) from available data. This removes the knowledge bottleneck in the traditional knowledge-driven decision making, which requires a strong support from domain experts. In this paper, we study data-driven decision making in the context of interactive dynamic influence diagrams (I-DIDs)—a general framework for multiagent sequential decision making under uncertainty. We propose a data-driven framework to solve the I-DIDs model and focus on learning the behavior of other agents in problem domains. The challenge is on learning a complete policy tree that will be embedded in the I-DIDs models due to limited data. We propose two new methods to develop complete policy trees for the other agents in the I-DIDs. The first method uses a simple clustering process, while the second one employs sophisticated statistical checks. We analyze the proposed algorithms in a theoretical way and experiment them over two problem domains.

Multi-Agent Deep Reinforcement Learning-Based Cooperative Edge Caching for Ultra-Dense Next-Generation Networks

The soaring mobile data traffic demands have spawned the innovative concept of mobile edge caching in ultra-dense next-generation networks, which mitigates their heavy traffic burden. We conceive cooperative content sharing between base stations (BSs) for improving the exploitation of the limited storage of a single edge cache. We formulate the cooperative caching problem as a partially observable Markov decision process (POMDP) based multi-agent decision problem, which jointly optimizes the costs of fetching contents from the local BS, from the nearby BSs and from the remote servers. To solve this problem, we devise a multi-agent actor-critic framework, where a communication module is introduced to extract and share the variability of the actions and observations of all BSs. To beneficially exploit the spatio-temporal differences of the content popularity, we harness a variational recurrent neural network (VRNN) for estimating the time-variant popularity distribution in each BS. Based on multi-agent deep reinforcement learning, we conceive a cooperative edge caching algorithm where the BSs operate cooperatively, since the distributed decision making of each agent depends on both the local and the global states. Our experiments conducted within a large scale cellular network having numerous BSs reveal that the proposed algorithm relying on the collaboration of BSs substantially improves the benefits of edge caches.

A Distributed Iterative Algorithm for Multi-Agent MILPs: Finite-Time Feasibility and Performance Characterization

We deal with decision making in a large-scale multi-agent system, where each agent aims at minimizing a local cost function subject to local constraints, and the local decision variables of all agents are coupled through a global constraint. We consider a cooperative framework where the multi-agent decision problem is formulated as a constrained optimization program with the sum of the local costs as global cost to be minimized with respect to the local decision variables of all agents, subject to both local and global constraints. We focus on a non-convex linear set-up where all costs and constraints are linear but local decision variables are discrete or include a discrete component, and propose a distributed iterative scheme based on dual decomposition and consensus to solve the resulting mixed integer linear program. Our approach extends recent results in the literature to a distributed set-up with a time-varying communication network and allows to: reduce the computational and communication effort, achieve resilience to communication failures, and also preserve privacy of local information. The approach is demonstrated on a numerical example of optimal charging of plug-in electric vehicles.

Impulsivity and information in cooperative and non-cooperative decision problems

Psychological evidences of impulsivity and false consensus eect lead results far from rationality. It is shown that impulsivity modies the discount function of each individual, and false consensus eect increases the degree of consensus in a multi-agent decision problem. Analyzing them together we note that in strategic interactions these two human factors involve choices which change equilibriums expected by rational individuals.

Dynamics of manipulation in voting, veto and plurality

Multi-agent decision problems, in which independent agents have to agree on a joint plan of action or allocation of resources, are central to artificial intelligence. The main focus of paper is the analysis of dynamics of manipulation in voting rules like plurality and veto. An important technical issue that arises is manipulation of voting schemes: a voter may be able to improve the outcome (with respect to his own preferences) by reporting his preferences incorrectly. We consider scenarios where voters cannot coordinate their actions, but are allowed to change their vote after observing the current outcome, as is often the case both in offline committees and in online voting. Voters are allowed to change their votes if they can get their desirable results, we have worked on veto and plurality rule with the small number of candidates and voters. We also used different moves for analysing the dynamics of voting system and concluded different results based on different types of moves (both manipulative and non-manipulative). We also defined a new tie breaking rule “Typicographical rule” and according to our observation it works better than the lexicographical rule.

Deep Reinforcement Learning for Event-Driven Multi-Agent Decision Processes

The incorporation of macro-actions (temporally extended actions) into multi-agent decision problems has the potential to address the curse of dimensionality associated with such decision problems. Since macro-actions last for stochastic durations, multiple agents executing decentralized policies in cooperative environments must act asynchronously. We present an algorithm that modifies generalized advantage estimation for temporally extended actions, allowing a state-of-the-art policy optimization algorithm to optimize policies in Dec-POMDPs in which agents act asynchronously. We show that our algorithm is capable of learning optimal policies in two cooperative domains, one involving real-time bus holding control and one involving wildfire fighting with unmanned aircraft. Our algorithm works by framing problems as "event-driven decision processes," which are scenarios in which the sequence and timing of actions and events are random and governed by an underlying stochastic process. In addition to optimizing policies with continuous state and action spaces, our algorithm also facilitates the use of event-driven simulators, which do not require time to be discretized into time-steps. We demonstrate the benefit of using event-driven simulation in the context of multiple agents taking asynchronous actions. We show that fixed time-step simulation risks obfuscating the sequence in which closely separated events occur, adversely affecting the policies learned. In addition, we show that arbitrarily shrinking the time-step scales poorly with the number of agents.

Iterative voting and acyclic games

Multi-agent decision problems, in which independent agents have to agree on a joint plan of action or allocation of resources, are central to artificial intelligence. In such situations, agents' individual preferences over available alternatives may vary, and may try to reconcile these differences by voting.We consider scenarios where voters cannot coordinate their actions, but are allowed to change their vote after observing the current outcome, as is often the case both in offline committees and in online voting. Specifically, we are interested in identifying conditions under which such iterative voting processes are guaranteed to converge to a Nash equilibrium state—that is, under which this process is acyclic. We classify convergence results based on the underlying assumptions about the agent scheduler (the order in which the agents take their actions) and the action scheduler (the actions available to the agents at each step). By so doing, we position iterative voting models within the general framework of acyclic games and game forms.In more detail, our main technical results provide a complete picture of conditions for acyclicity in several variations of Plurality voting. In particular, we show that (a) under the traditional lexicographic tie-breaking, the game converges from any state and for any order of agents, under a weak restriction on voters' actions; and that (b) Plurality with randomized tie-breaking is not guaranteed to converge under arbitrary agent schedulers, but there is always some path of better replies from any initial state of the game to a Nash equilibrium. We thus show a first separation between order-free acyclicity and weak acyclicity of game forms, thereby settling an open question from [7]. In addition, we refute another conjecture of Kukushkin regarding strongly acyclic voting rules, by proving the existence of strongly acyclic separable game forms.

Evaluating the Robustness of Game Theoretic Solutions When Using Abstraction

Game theory is a tool for modeling multi-agent decision problems and has been used to analyze strategies in domains such as poker, security, and trading agents. One method for solving very large games is to use abstraction techniques to shrink the game by removing detail, solve the reduced game, and then translate the solution back to the original game. We present a methodology for evaluating the robustness of different game-theoretic solution concepts to the errors introduced by the abstraction process. We present an initial empirical study of the robustness of several solution methods when using abstracted games.

The intertemporal choice behaviour: The role of emotions in a multi-agent decision problem

Decision Neuroscience has shown positive and negative side of emotions in intertemporal choices. Psychological evidences, indeed, point out anomalies (impulsivity modifies the discount function of each individual) and the false consensus effect which increases the degree of consensus in a multi-agent decision problem. An experiment (Engelmann and Strobel 2004) demonstrates that the relevance of the false consensus effect depends on the difficulty of the information retrieval, so the underlying mechanism is an information processing deficiency rather than egocentricity. We demonstrate that emotions can not cause anomalies in a cooperative strategic interaction because information is explicit.

Opportunistic Approachability and Generalized No-Regret Problems

Blackwell’s theory of approachability, introduced in 1956, has since proved a useful tool in the study of a range of repeated multiagent decision problems. Given a repeated matrix game with vector payoffs, a target set S is approachable by a certain player if he can ensure that the average payoff vector converges to that set, for any strategy of the opponent. In this paper we consider the case where a set need not be approachable in general, but may be approached if the opponent played favorably in some sense. In particular, we consider nonconvex sets that satisfy Blackwell’s dual condition, namely, can be approached when the opponent plays a stationary strategy. Whereas the convex hull of such a set is approachable, this is not generally the case for the original nonconvex set itself. We start by defining a sense of restricted play of the opponent (with stationary strategies being a special case), and then formulate appropriate goals for an opportunistic approachability algorithm that can take advantage of such restricted play as it unfolds during the game. We then consider a calibration-based approachability strategy that is opportunistic in that sense. A major motivation for this study comes from no-regret problems that lack a convex structure such as the problem of online learning with sample-path constraints, as formulated in Mannor et al. [Mannor S, Tsitsiklis JN, Yu JY (2009) Online learning with sample path constraints. J. Machine Learn. Res. 10:569–590]. Here the best-response-in-hindsight is not generally attainable, but only a convex relaxation thereof. Our proposed algorithm, while ensuring that relaxed goal, also comes closer to the nonrelaxed one when the opponent’s play is restricted in a well-defined sense.

Decentralized strategy selection with learning automata for multiple pursuer–evader games

The multiple pursuers and evaders game may be represented as a Markov game. Using this modeling, one may interpret each player as a decentralized unit that has to work independently in order to complete a task. This is a distributed multiagent decision problem and several different possible solutions have already been proposed. However, most solutions require some sort of central coordination. In this paper, we intend to model each player as a learning automaton and let them evolve and adapt in order to solve the difficult problem they have at hand. We are also going to show that, using the proposed learning process, the players’ policies will converge to an equilibrium point. Simulations of such scenarios with multiple pursuers and evaders are presented in order to show the feasibility of the approach.

Automated Action Abstraction of Imperfect Information Extensive-Form Games

Multi-agent decision problems can often be formulated as extensive-form games. We focus on imperfect information extensive-form games in which one or more actions at many decision points have an associated continuous or many-valued parameter. A stock trading agent, in addition to deciding whether to buy or not, must decide how much to buy. In no-limit poker, in addition to selecting a probability for each action, the agent must decide how much to bet for each betting action. Selecting values for these parameters makes these games extremely large. Two-player no-limit Texas Hold'em poker with stacks of 500 big blinds has approximately 1071 states, which is more than 1050 times more states than two-player limit Texas Hold'em. The main contribution of this paper is a technique that abstracts a game's action space by selecting one, or a small number, of the many values for each parameter. We show that strategies computed using this new algorithm for no-limit Leduc poker exhibit significant utility gains over epsilon-Nash equilibrium strategies computed with standard, hand-crafted parameter value abstractions.

Solving efficiently Decentralized MDPs with temporal and resource constraints

Optimizing the operation of cooperative multi-agent systems that can deal with large and realistic problems has become an important focal area of research in the multi-agent community. In this paper, we first present a new model, the OC-DEC-MDP (Opportunity Cost Decentralized Markov Decision Process), that allows us to represent large multi-agent decision problems with temporal and precedence constraints. Then, we propose polynomial algorithms to efficiently solve problems formalized by OC-DEC-MDPs. The problems we deal with consist of a set of agents that have to execute a set of tasks in a cooperative way. The agents cannot communicate during task execution and they must respect resource and temporal constraints. Our approach is based on Decentralized Markov Decision Processes (DEC-MDPs) and uses the concept of opportunity cost borrowed from economics to obtain approximate control policies. Experimental results show that our approach produces good quality solutions for complex problems which are out of reach of existing approaches.

Convergence to Equilibria in Plurality Voting

Multi-agent decision problems, in which independent agents have to agree on a joint plan of action or allocation of resources, are central to AI. In such situations, agents' individual preferences over available alternatives may vary, and they may try to reconcile these differences by voting. Based on the fact that agents may have incentives to vote strategically and misreport their real preferences, a number of recent papers have explored different possibilities for avoiding or eliminating such manipulations. In contrast to most prior work, this paper focuses on convergence of strategic behavior to a decision from which no voter will want to deviate. We consider scenarios where voters cannot coordinate their actions, but are allowed to change their vote after observing the current outcome. We focus on the Plurality voting rule, and study the conditions under which this iterative game is guaranteed to converge to a Nash equilibrium (i.e., to a decision that is stable against further unilateral manipulations). We show for the first time how convergence depends on the exact attributes of the game, such as the tie-breaking scheme, and on assumptions regarding agents' weights and strategies.

Coordinated learning in multiagent MDPs with infinite state-space

In this paper we address the problem of simultaneous learning and coordination in multiagent Markov decision problems (MMDPs) with infinite state-spaces. We separate this problem in two distinct subproblems: learning and coordination. To tackle the problem of learning, we survey Q-learning with soft-state aggregation (Q-SSA), a well-known method from the reinforcement learning literature (Singh et al. in Advances in neural information processing systems. MIT Press, Cambridge, vol 7, pp 361---368, 1994). Q-SSA allows the agents in the game to approximate the optimal Q-function, from which the optimal policies can be computed. We establish the convergence of Q-SSA and introduce a new result describing the rate of convergence of this method. In tackling the problem of coordination, we start by pointing out that the knowledge of the optimal Q-function is not enough to ensure that all agents adopt a jointly optimal policy. We propose a novel coordination mechanism that, given the knowledge of the optimal Q-function for an MMDP, ensures that all agents converge to a jointly optimal policy in every relevant state of the game. This coordination mechanism, approximate biased adaptive play (ABAP), extends biased adaptive play (Wang and Sandholm in Advances in neural information processing systems. MIT Press, Cambridge, vol 15, pp 1571---1578, 2003) to MMDPs with infinite state-spaces. Finally, we combine Q-SSA with ABAP, this leading to a novel algorithm in which learning of the game and coordination take place simultaneously. We discuss several important properties of this new algorithm and establish its convergence with probability 1. We also provide simple illustrative examples of application.

MULTI-AGENT GRAPHICAL DECISION MODELS IN MEDICINE

Many practical applications in the medical domain require a cooperative decision with multiple entities (agents). These applications are instances of a multi-agent decision problem. This complex decision problem often concerns a large knowledge domain and involves some agency properties. It disables traditional methods on probabilistic graphical decision models. In this article, we propose a new representation including multiply sectioned influence diagrams (MSIDs) and hyper relevance graphs (HRGs). An MSID represents decision problems involving multiple agents in a distributed and flexible fashion, while an HRG encodes organizational relationships in a multi-agent system. Subsequently, a symbolic method is extended to facilitate the model verification with the aim of building a valid decision model. An evaluation algorithm based on the junction tree algorithm is developed to solve an MSID. Some relevant evaluation strategies are analyzed. The decision problem on the Severe Acute Respiratory Syndrome (SARS) control is illustrated with our proposed methodologies throughout this article.