Decentralized Markov Decision Processes Research Articles

Coordinated Multi-Robot Exploration Under Communication Constraints Using Decentralized Markov Decision Processes

Recent works on multi-agent sequential decision making using decentralized partially observable Markov decision processes have been concerned with interaction-oriented resolution techniques and provide promising results. These techniques take advantage of local interactions and coordination. In this paper, we propose an approach based on an interaction-oriented resolution of decentralized decision makers. To this end, distributed value functions (DVF) have been used by decoupling the multi-agent problem into a set of individual agent problems. However existing DVF techniques assume permanent and free communication between the agents. In this paper, we extend the DVF methodology to address full local observability, limited share of information and communication breaks. We apply our new DVF in a real-world application consisting of multi-robot exploration where each robot computes locally a strategy that minimizes the interactions between the robots and maximizes the space coverage of the team even under communication constraints. Our technique has been implemented and evaluated in simulation and in real-world scenarios during a robotic challenge for the exploration and mapping of an unknown environment. Experimental results from real-world scenarios and from the challenge are given where our system was vice-champion.

UAV Formation Shape Control via Decentralized Markov Decision Processes

In this paper, we present a decentralized unmanned aerial vehicle (UAV) swarm formation control approach based on a decision theoretic approach. Specifically, we pose the UAV swarm motion control problem as a decentralized Markov decision process (Dec-MDP). Here, the goal is to drive the UAV swarm from an initial geographical region to another geographical region where the swarm must form a three-dimensional shape (e.g., surface of a sphere). As most decision-theoretic formulations suffer from the curse of dimensionality, we adapt an existing fast approximate dynamic programming method called nominal belief-state optimization (NBO) to approximately solve the formation control problem. We perform numerical studies in MATLAB to validate the performance of the above control algorithms.

An extended study on addressing defender teamwork while accounting for uncertainty in attacker defender games using iterative Dec-MDPs

Multi-agent teamwork and defender-attacker security games are two areas that are currently receiving significant attention within multi-agent systems research. Unfortunately, despite the need for effective teamwork among multiple defenders, little has been done to harness the teamwork research in s ecurity games. The problem that this paper seeks to solve is the coordination of decentralized defender agents in the presence of uncertainty while securing targets against an observing adversary. To address this problem, we offer the following novel contributions in this paper: (i) New model of security games with defender teams that coordinate under uncertainty; (ii) New algorithm based on column generation that utilizes Decentralized Markov Decision Processes (Dec-MDPs) to generate defender strategies that incorporate uncertainty; (iii) New techniques to handle global events (when one or more agents may leave the system) during defender execution; (iv) Heuristics that help scale up in the number of targets and agents to handle real-world scenarios; (v) Exploration of the robustness of randomized pure strategies. The paper opens the door to a potentially new area combining computational game theory and multi-agent teamwork.

An extended version of opportunity cost algorithm for communication decisions

Decentralized Markov decision processes (DEC-MDPs) provide powerful modeling tools for cooperative multi-agent decision making under uncertainty. In this paper, we tackle particular subclasses of theoretic decision models which operate under time pressure having uncertain actions’ durations. Particularly, we extend a solution method called opportunity cost decentralized Markov decision process (OC-DEC-MDP) to handle more complex precedence constraints where actions of each agent are presented by a partial plan. As a result of local partial plans with precedence constraints between agents, mis-coordination situations may occur. For this purpose, we introduce communication decisions between agents. Since dealing with offline planning for communication increase state space size, we aim at restricting the use of communication. To this end, we propose to exploit problem structure in order to limit communication decisions. Moreover, we study two separate cases about the reliability of the communication. The first case we assume that the communication is always successful (i.e. all messages are always successfully received). The second case, we enhance our policy computation algorithm to deal with possibly missed messages. Experimental results show that even if communication is costly, it improves the degree of coordination between agents and it increases team performances regarding constraints.

A DEC-MDP model for joint uplink/downlink resource management in OFDMA-based networks

With the advent of mobile services with asymmetric and symmetric quality of service (QoS) requirements, traditional single link resource allocation techniques have started to show some limitations in handling the complex requirements. To address these issues, joint uplink/downlink resource management approaches were recently introduced where both communications links are jointly considered in the resource management process. One direct consequence of this coupling is a modification of the underlying queueing behavior since the decision making process in one direction in terms of transmission rate now depends on the performance achieved in the opposite direction. In this paper, we present a modeling approach of the decision making process that takes place under the joint uplink/downlink resource management framework. Using decentralized Markov decision processes (DEC-MDP) as a model and gradient ascent methods as an optimization technique, we formulate and solve the joint uplink/downlink decision making process. The uplink and downlink of each user are considered as agents. Assuming certain subcarrier and power allocation schemes, we investigate the resource usage in the uplink and downlink to achieve a certain delay balancing constraint where the total delay in the uplink and downlink is bound by a pre-determined threshold. The approach followed starts by modeling the problem in hand using DEC-MDPs. After discussing the different aspects of the model, the solution using gradient ascent is described. Simulation results illustrate the different dimensions of the problem and their impact on the resource management process.

Solving efficiently Decentralized MDPs with temporal and resource constraints

Optimizing the operation of cooperative multi-agent systems that can deal with large and realistic problems has become an important focal area of research in the multi-agent community. In this paper, we first present a new model, the OC-DEC-MDP (Opportunity Cost Decentralized Markov Decision Process), that allows us to represent large multi-agent decision problems with temporal and precedence constraints. Then, we propose polynomial algorithms to efficiently solve problems formalized by OC-DEC-MDPs. The problems we deal with consist of a set of agents that have to execute a set of tasks in a cooperative way. The agents cannot communicate during task execution and they must respect resource and temporal constraints. Our approach is based on Decentralized Markov Decision Processes (DEC-MDPs) and uses the concept of opportunity cost borrowed from economics to obtain approximate control policies. Experimental results show that our approach produces good quality solutions for complex problems which are out of reach of existing approaches.

Algorithmes de co-évolution pour la résolution approchée de PDM multi-agent

Markov Decision Processes (MDP) provide a formal approach for solving sequential decision problems under uncertainty. Decentralized Markov Decision Processes extend MDP. They are used to model the problem of several agents making decision under uncertainty. However, solving a decentralized markov decision process is NEXP-complete as soon as two agents are involved. In this paper we propose algorithms for solving approximately Multi-agent MDP problems in a centralized or decentralized way. The aim of our methods is to design a system in which agents coordinate to achieve a task in collaboration. Coordination is based on two major properties: subjectivity and empathy. While subjectivity allows an agent to deal with incomplete and local perceptions, thus to design memory less policies, empathy allows an agent to adapt its decisions in regards with the uncertainty of the decisions of the others.

Solving Transition Independent Decentralized Markov Decision Processes

Formal treatment of collaborative multi-agent systems has been lagging behind the rapid progress in sequential decision making by individual agents. Recent work in the area of decentralized Markov Decision Processes (MDPs) has contributed to closing this gap, but the computational complexity of these models remains a serious obstacle. To overcome this complexity barrier, we identify a specific class of decentralized MDPs in which the agents' transitions are independent. The class consists of independent collaborating agents that are tied together through a structured global reward function that depends on all of their histories of states and actions. We present a novel algorithm for solving this class of problems and examine its properties, both as an optimal algorithm and as an anytime algorithm. To our best knowledge, this is the first algorithm to optimally solve a non-trivial subclass of decentralized MDPs. It lays the foundation for further work in this area on both exact and approximate algorithms.

Solving transition independent decentralized Markov decision processes

Formal treatment of collaborative multi-agent systems has been lagging behind the rapid progress in sequential decision making by individual agents. Recent work in the area of decentralized Markov Decision Processes (MDPs) has contributed to closing this gap, but the computational complexity of these models remains a serious obstacle. To overcome this complexity barrier, we identify a specific class of decentralized MDPs in which the agents' transitions are independent. The class consists of independent collaborating agents that are tied together through a structured global reward function that depends on all of their histories of states and actions. We present a novel algorithm for solving this class of problems and examine its properties, both as an optimal algorithm and as an anytime algorithm. To the best of our knowledge, this is the first algorithm to optimally solve a non-trivial subclass of decentralized MDPs. It lays the foundation for further work in this area on both exact and approximate algorithms.