Multi-agent Domains Research Articles

Information Theoretic Approach for Measuring Interaction in Multiagent Domain

Although a large number of reinforcement learning algorithms have been proposed for the generation of cooperative behaviors, the question of how to evaluate mutual benefit or loss among them is still open. As far as we know, an emerged behavior is regarded as a cooperative behavior when embedded agents have finally achieved their global goal, regardless of whether or not mutual interference has had any effect during the course of the learning process of each agent. Thus, we cannot detect any harmful interaction on the way to achieving a fully-converged policy. In this paper, we propose a measure based on information theory for evaluating the degree of interaction during the learning process from the viewpoint of information sharing. In order to discuss the bad effects of concurrent learning, we apply our proposed measure to a situation in which there exist conflicts among the agents, and we show the availability of our measure.

Time Horizon Generalization in Reinforcement Learning: Generalizing Multiple Q-Tables in Q-Learning Agents

This paper focuses on generalization in reinforcement learning from the time horizon viewpoint, exploring the method that generalizes multiple Q-tables in the multiagent reinforcement learning domain. For this purpose, we propose time horizon generalization for reinforcement learning, which consists of (1) Q-table selection method and (2) Q-table merge timing method, enabling agents to (1) select which Q-tables can be generalized from among many Q-tables and (2) determine when the selected Q-tables should be generalized. Intensive simulation on the bargaining game as sequential interaction game have revealed the following implications: (1) both Q-table selection and merging timing methods help replicate the subject experimental results without ad-hoc parameter setting; and (2) such replication succeeds by agents using the proposed methods with smaller numbers of Q-tables.

Multi-agent coordination by temporal plan fusion: Application to combat search and rescue

Coordination of actions and plans that must be achieved by multiple agents is one of the most difficult tasks in the multi-agent domain. In order to work together and achieve a common goal, agents need to coordinate their plans in a way that guarantees, if possible, the success of each individual agent plan. In this work, we propose a temporal fusion mechanism that allows a set of agents to fuse their plans and generate a global coordinated plan. First, we define a temporal plan as a set of temporally constrained actions. The fusion of several temporal plans is a temporal plan, which can be executed by several agents. The proposed framework is applied to a Combat Search and Rescue application.

Effort Games and the Price of Myopia

AbstractWe consider Effort Games, a game‐theoretic model of cooperation in open environments, which is a variant of the principal‐agent problem from economic theory. In our multiagent domain, a common project depends on various tasks; carrying out certain subsets of the tasks completes the project successfully, while carrying out other subsets does not. The probability of carrying out a task is higher when the agent in charge of it exerts effort, at a certain cost for that agent. A central authority, called the principal, attempts to incentivize agents to exert effort, but can only reward agents based on the success of the entire project.We model this domain as a normal form game, where the payoffs for each strategy profile are defined based on the different probabilities of carrying out each task and on the boolean function that defines which task subsets complete the project, and which do not. We view this boolean function as a simple coalitional game, and call this game the underlying coalitional game. We suggest the Price of Myopia (PoM) as a measure of the influence the model of rationality has on the minimal payments the principal has to make in order to motivate the agents in such a domain to exert effort.We consider the computational complexity of testing whether exerting effort is a dominant strategy for an agent, and of finding a reward strategy for this domain, using either a dominant strategy equilibrium or using iterated elimination of dominated strategies. We show these problems are generally #P‐hard, and that they are at least as computationally hard as calculating the Banzhaf power index in the underlying coalitional game. We also show that in a certain restricted domain, where the underlying coalitional game is a weighted voting game with certain properties, it is possible to solve all of the above problems in polynomial time. We give bounds on PoM in weighted voting effort games, and provide simulation results regarding PoM in another restricted class of effort games, namely effort games played over Series‐Parallel Graphs (© 2009 WILEY‐VCH Verlag GmbH & Co. KGaA, Weinheim)

COOPERATIVE LEARNING BY POLICY-SHARING IN MULTIPLE AGENTS

Reinforcement learning is one of the more prominent machine-learning technologies due to its unsupervised learning structure and ability to continually learn, even in a dynamic operating environment. Applying this learning to cooperative multi-agent systems not only allows each individual agent to learn from its own experience, but also offers the opportunity for the individual agents to learn from the other agents in the system so the speed of learning can be accelerated. In the proposed learning algorithm, an agent adapts to comply with its peers by learning carefully when it obtains a positive reinforcement feedback signal, but should learn more aggressively if a negative reward follows the action just taken. These two properties are applied to develop the proposed cooperative learning method. This research presents the novel use of the fastest policy hill-climbing methods of Win or Lose Fast (WoLF) with policy-sharing. Results from the multi-agent cooperative domain illustrate that the proposed algorithms perform better than Q-learning alone in a piano mover environment. It also demonstrates that agents can learn to accomplish a task together efficiently through repetitive trials.

Approximating power indices: theoretical and empirical analysis

Many multiagent domains where cooperation among agents is crucial to achieving a common goal can be modeled as coalitional games. However, in many of these domains, agents are unequal in their to affect the outcome of the game. Prior research on weighted voting games has explored indices, which reflect how much real power a voter has. Although primarily used for voting games, these indices can be applied to any simple coalitional game. Computing these indices is known to be computationally hard in various domains, so one must sometimes resort to approximate methods for calculating them. We suggest and analyze randomized methods to approximate indices such as the Banzhaf index and the Shapley---Shubik index. Our approximation algorithms do not depend on a specific representation of the game, so they can be used in any simple coalitional game. Our methods are based on testing the game's value for several sample coalitions. We show that no approximation algorithm can do much better for general coalitional games, by providing lower bounds for both deterministic and randomized algorithms for calculating indices. We also provide empirical results regarding our method, and show that it typically achieves much better accuracy and confidence than those required.

An online multi-agent co-operative learning algorithm in POMDPs

Solving partially observable Markov decision processes (POMDPs) is a complex task that is often intractable. This paper examines the problem of finding an optimal policy for POMDPs. While a lot of effort has been made to develop algorithms to solve POMDPs, the question of automatically finding good low-dimensional spaces in multi-agent co-operative learning domains has not been explored thoroughly. To identify this question, an online algorithm CMEAS is presented to improve the POMDP model. This algorithm is based on a look-ahead search to find the best action to execute at each cycle. Thus the overwhelming complexity of computing a policy for each possible situation is avoided. A series of simulations demonstrate this good strategy and performance of the proposed algorithm when multiple agents co-operate to find an optimal policy for POMDPs.

Solving Multiagent Networks Using Distributed Constraint Optimization

In many cooperative multiagent domains, the effect of local interactions between agents can be compactly represented as a network structure. Given that agents are spread across such a network, agents directly interact only with a small group of neighbors. A distributed constraint optimization problem (DCOP) is a useful framework to reason about such networks of agents. Given agents’ inability to communicate and collaborate in large groups in such networks, we focus on an approach called k-optimality for solving DCOPs. In this approach, agents form groups of one or more agents until no group of k or fewer agents can possibly improve the DCOP solution; we define this type of local optimum, and any algorithm guaranteed to reach such a local optimum, as k-optimal. The article provides an overview of three key results related to koptimality. The first set of results gives worst-case guarantees on the solution quality of k-optima in a DCOP. These guarantees can help determine an appropriate k-optimal algorithm, or possibly an appropriate constraint graph structure, for agents to use in situations where the cost of coordination between agents must be weighed against the quality of the solution reached. The second set of results gives upper bounds on the number of k-optima that can exist in a DCOP. These results are useful in domains where a DCOP must generate a set of solutions rather than a single solution. Finally, we sketch algorithms for k-optimality and provide some experimental results for 1-, 2- and 3-optimal algorithms for several types of DCOPs.

Verifiable agent interaction in abductive logic programming

SCIFF is a framework thought to specify and verify interaction in open agent societies. The SCIFF language is equipped with a semantics based on abductive logic programming; SCIFF's operational component is a new abductive logic programming proof procedure, also named SCIFF, for reasoning with expectations in dynamic environments. In this article we present the declarative and operational semantics of the SCIFF language, and the termination, soundness, and completeness results of the SCIFF proof procedure, and we demonstrate SCIFF's possible application in the multiagent domain.

A knowledge-based tool for multi-agent domain engineering

Ontologies provide an unambiguous terminology that can be shared by all involved in a software development process. They can also be as generic as needed allowing its reuse and easy extension. These features turn ontologies useful for representing the knowledge of software engineering techniques and methodologies, and an appropriate abstraction mechanism for the specification of high-level reusable software artifacts like domain models, frameworks and software patterns. This work describes ONTOMADEM, a conceptualization of MADEM (“multi-agent domain engineering methodology”), a software development methodology for multi-agent domain Engineering. The ontology is used as a knowledge-based tool for capturing and representing the products of a multi-agent domain engineering process, created through the instantiation of its hierarchy of classes.

MALEF: Framework for distributed machine learning and data mining

Growing importance of distributed data mining techniques has recently attracted attention of researchers in multiagent domain. In this paper we present a novel framework MultiAgent Learning Framework (MALEF) designed for both the agent-based distributed machine learning as well as data mining. Proposed framework is based on: the exchange of meta-level descriptions of individual learning process; online reasoning about learning success and learning progress. This paper illustrates how MALEF framework can be used in practical system in which different learners use different datasets, hypotheses and learning algorithms. We describe our experimental results obtained using this system and review related work on the subject.

FUZZY STATE AGGREGATION AND POLICY HILL CLIMBING FOR STOCHASTIC ENVIRONMENTS

Reinforcement learning is one of the more attractive machine learning technologies, due to its unsupervised learning structure and ability to continually learn even as the operating environment changes. Additionally, by applying reinforcement learning to multiple cooperative software agents (a multi-agent system) not only allows each individual agent to learn from its own experience, but also opens up the opportunity for the individual agents to learn from the other agents in the system, thus accelerating the rate of learning. This research presents the novel use of fuzzy state aggregation, as the means of function approximation, combined with the fastest policy hill climbing methods of Win or Lose Fast (WoLF) and policy-dynamics based WoLF (PD-WoLF). The combination of fast policy hill climbing and fuzzy state aggregation function approximation is tested in two stochastic environments: Tileworld and the simulated robot soccer domain, RoboCup. The Tileworld results demonstrate that a single agent using the combination of FSA and PHC learns quicker and performs better than combined fuzzy state aggregation and Q-learning reinforcement learning alone. Results from the multi-agent RoboCup domain again illustrate that the policy hill climbing algorithms perform better than Q-learning alone in a multi-agent environment. The learning is further enhanced by allowing the agents to share their experience through a weighted strategy sharing.

A domain model of Web recommender systems based on usage mining and collaborative filtering

Considering the increasing demand of multi-agent systems, the practice of software reuse is essential to the development of such systems. Multi-agent domain engineering is a process for the construction of domain-specific agent-based reusable software artifacts, like domain models, representing the requirements of a family of multi-agent systems in a domain, and frameworks, implementing reusable agent-based design solutions to those requirements. This article describes the domain modeling tasks of the MADEM methodology and a case study on the application of GRAMO, a MADEM technique, for the construction of the domain model of ONTOWUM, specifying the common and variable requirements of a family of Web recommender systems based on usage mining and collaborative filtering.

Coordinated Multiagent Teams and Sliding Autonomy for Large-Scale Assembly

Recent research in human-robot interaction has investigated the concept of Sliding, or Adjustable, Autonomy, a mode of operation bridging the gap between explicit teleoperation and complete robot autonomy. This work has largely been in single-agent domains-involving only one human and one robot-and has not examined the issues that arise in multiagent domains. We discuss the issues involved in adapting Sliding Autonomy concepts to coordinated multiagent teams. In our approach, remote human operators have the ability to join, or leave, the team at will to assist the autonomous agents with their tasks (or aspects of their tasks) while not disrupting the team's coordination. Agents model their own and the human operator's performance on subtasks to enable them to determine when to request help from the operator. To validate our approach, we present the results of two experiments. The first evaluates the human/multirobot team's performance under four different collaboration strategies including complete teleoperation, pure autonomy, and two distinct versions of Sliding Autonomy. The second experiment compares a variety of user interface configurations to investigate how quickly a human operator can attain situational awareness when asked to help. The results of these studies support our belief that by incorporating a remote human operator into multiagent teams, the team as a whole becomes more robust and efficient

Coach planning with opponent models for distributed execution

In multi-agent domains, the generation and coordinated execution of plans in the presence of adversaries is a significant challenge. In our research, a special agent works with a team of distributed agents. The coach has a global view of the world, but has no actions other than occasionally communicating with the team over a limited bandwidth channel. Our coach is given a set of predefined opponent models which predict future states of the world caused by the opponents' actions. The coach observes the world state changes resulting from the execution of its team and opponents and selects the best matched opponent model based on its observations. The coach uses the recognized opponent model to predict the behavior of the opponent. Upon opportunities to communicate, the coach generates a plan for the team, using the predictions of the opponent model. The centralized coach generates a plan for distributed execution. We introduce (i) the probabilistic representation and recognition algorithm for the opponent models; (ii) a multi-agent plan representation, Multi-Agent Simple Temporal Networks; and (iii) a plan execution algorithm that allows the robust distributed execution in the presence of noisy perception and actions. The complete approach is implemented in a complex simulated robot soccer environment. We present the contributions as developed in this domain, carefully highlighting their generality along with a series of experiments validating the effectiveness of our coach approach.

Experimental Analysis of Reward Design for Continuing Task in Multiagent Domains -- RoboCup Soccer Keepaway --

In this paper, we discuss guidelines for a reward design problem that defines when and what amount of reward should be given to the agent/s, within the context of reinforcement learning approach. We would like to take keepaway soccer as a standard task of the multiagent domain which requires skilled teamwork. The difficulties of designing reward for this task are due to its features as follows: i) since it belongs to the continuing task which has no explicit goal to achieve, it is hard to tell when reward should be given to the agent/s. ii) since it is a multiagent cooperative task, it is hard to decide what is a fair share of reward for each agent's contribution to achieve the goal. Through some experiments, we show that the reward design have a major effect on the agent's behavior, and introduce the successful reward function that makes agents perform keepaway better and more interesting than the conventional one does. Finally, we explore the relationship between `reward design' and `acquired behaviors' from the viewpoint of teamwork.

MULTI-AGENT-BASED SUPPORT FOR ELECTRONIC CONTRACTING IN VIRTUAL ENTERPRISES

In this paper we present multi-agent based management of electronic contract (e-contract) in a B2B context, for Tele-services. Several solutions have been proposed in the multi-agent domain for negotiation of contracts. However few concern the monitoring of contracts and the management of their violation and enforcement. We describe a prototype implementing the concepts of e-contract using ***ℳOISEInst, a normative multi-agent organizational model. Thanks to it, contractual clauses are expressed as rights and duties of agents along four points of view: structural, functional, contextual and normative. We use a real-world contracting scenario as a test-bed for examining the proposed e-contract architecture and for implementing our prototype.

A system of agent-based software patterns for user modeling based on usage mining

In adaptive hypermedia systems, a user can select explicitly an adaptation effect or he/she can leave the system execute some of these functions. An important component of an adaptive system is the ability to model the users of the system according to their goals and preferences. Web usage mining aims at discover interesting patterns of use by analyzing Web usage data. This information can be used to capture implicitly user models and used them for the adaptation of systems. User modeling and system adaptability can be approached through the agent paradigm. This article summarizes a system of architectural and detailed design patterns describing known agent-based solutions to recurrent problems of user modeling based on usage mining along with the description of a general purpose problem-solving architectural pattern used by some of the first ones. Patterns are derived from recurrent designs of specific agent-based applications. The proposed patterns are being developed in the context of a Multi-Agent Domain Engineering research project, which approaches software complexity and productivity through the construction of techniques and tools promoting software reuse in Multi-Agent Domain Engineering.

Rights and Argumentation in Open Multi-Agent Systems

As utility calculus cannot account for an important part of agents' behaviour in Multi-Agent Systems, researchers have progressively adopted a more normative approach. Unfortunately, social laws have turned out to be too restrictive in real-life domains where autonomous agents' activity cannot be completely specified in advance. It seems that a halfway concept between anarchic and off-line constrained interaction is needed. We think that the concept of right suits this idea. Rights improve coordination and facilitate social action in Multi-Agent domains. Rights allow the agents enough freedom, and at the same time constrain them (prohibiting specific actions). Besides, rights can be understood as the basic concept underneath open normative systems where the agents reason about the code they must abide by. Typically, in such systems this code is underspecified. On the other hand, the agents might not have complete knowledge about the rules governing their interaction. Conflict situations arise, thus, when the agents have different points of view as to how to apply the code. We have extended Parsons's et al. argumentation protocol (Parsons et al. 1998a, b) to normative systems to deal with this problem.

The RoboCup Soccer Server and CMUnited Clients: Implemented Infrastructure for MAS Research

The RoboCup Soccer Server and associated client code is a growing body of software infrastructure that enables a wide variety of multiagent systems research. The Soccer Server is a multiagent environment that supports 22 independent agents interacting in a complex, real-time environment. AI researchers have been using the Soccer Server to pursue research in a wide variety of areas, including real-time multiagent planning, real-time communication methods, collaborative sensing, and multiagent learning. This article describes the current Soccer Server and the champion CMUnited soccer-playing agents, both of which are publically available and used by a growing research community. It also describes the ongoing development of FUSS, a new, flexible simulation environment for multiagent research in a variety of multiagent domains.