Multi-agent Problem Research Articles

Solution Concepts in Hierarchical Games Under Bounded Rationality With Applications to Autonomous Driving

With autonomous vehicles (AV) set to integrate further into regular human traffic, there is an increasing consensus of treating AV motion planning as a multi-agent problem. However, the traditional game theoretic assumption of complete rationality is too strong for the purpose of human driving, and there is a need for understanding human driving as a bounded rational activity through a behavioral game theoretic lens. To that end, we adapt three metamodels of bounded rational behavior; two based on Quantal level-k and one based on Nash equilibria with quantal errors. We formalize the different solution concepts that can be applied in the context of hierarchical games, a framework used in multi-agent motion planning, for the purpose of creating game theoretic models of driving behavior. Furthermore, based on a contributed dataset of human driving at a busy urban intersection with a total of ~4k agents and ~44k decision points, we evaluate the behavior models on the basis of model fit to naturalistic data, as well as their predictive capacity. Our results suggest that among the behavior models evaluated, modeling driving behavior as pure strategy Nash equilibria with quantal errors at the level of maneuvers with bounds sampling of actions at the level of trajectories provides the best fit to naturalistic driving behavior, and there is a significant impact of situational factors on the performance of behavior models.

A Faithful Mechanism for Incremental Multi-Agent Agreement Problems with Self-Interested and Privacy-Preserving Agents

Distributed multi-agent agreement problems (MAPs) are central to many multi-agent systems. However, to date, the issues associated with encounters between self-interested and privacy-preserving agents have received limited attention. Given this, we develop the first distributed negotiation mechanism that enables self-interested agents to reach a socially desirable agreement with limited information leakage. The agents’ optimal negotiation strategies in this mechanism are investigated. Specifically, we propose a reinforcement learning-based approach to train agents to learn their optimal strategies in the proposed mechanism. Also, a heuristic algorithm is designed to find close-to-optimal negotiation strategies with reduced computational costs. We demonstrate the effectiveness and strength of our proposed mechanism through both game theoretical and numerical analysis. We prove theoretically that the proposed mechanism is budget balanced and motivates the agents to participate and follow the rules faithfully. The experimental results confirm that the proposed mechanism significantly outperforms the current state of the art, by increasing the social-welfare and decreasing the privacy leakage.

On the Linear Convergence of Two Decentralized Algorithms

Decentralized algorithms solve multi-agent problems over a connected network, where the information can only be exchanged with the accessible neighbors. Though there exist several decentralized optimization algorithms, there are still gaps in convergence conditions and rates between decentralized and centralized algorithms. In this paper, we fill some gaps by considering two decentralized algorithms: EXTRA and NIDS. They both converge linearly with strongly convex objective functions. We will answer two questions regarding them. What are the optimal upper bounds for their stepsizes? Do decentralized algorithms require more properties on the functions for linear convergence than centralized ones? More specifically, we relax the required conditions for linear convergence of both algorithms. For EXTRA, we show that the stepsize is comparable to that of centralized algorithms. For NIDS, the upper bound of the stepsize is shown to be exactly the same as the centralized ones. In addition, we relax the requirement for the objective functions and the mixing matrices. We provide the linear convergence results for both algorithms under the weakest conditions.

Multiagent Reinforcement Learning: Rollout and Policy Iteration

We discuss the solution of complex multistage decision problems using methods that are based on the idea of policy iteration (PI), i.e., start from some base policy and generate an improved policy. Rollout is the simplest method of this type, where just one improved policy is generated. We can view PI as repeated application of rollout, where the rollout policy at each iteration serves as the base policy for the next iteration. In contrast with PI, rollout has a robustness property: it can be applied on-line and is suitable for on-line replanning. Moreover, rollout can use as base policy one of the policies produced by PI, thereby improving on that policy. This is the type of scheme underlying the prominently successful AlphaZero chess program. In this paper we focus on rollout and PI-like methods for problems where the control consists of multiple components each selected (conceptually) by a separate agent. This is the class of multiagent problems where the agents have a shared objective function, and a shared and perfect state information. Based on a problem reformulation that trades off control space complexity with state space complexity, we develop an approach, whereby at every stage, the agents sequentially (one-at-a-time) execute a local rollout algorithm that uses a base policy, together with some coordinating information from the other agents. The amount of total computation required at every stage grows linearly with the number of agents. By contrast, in the standard rollout algorithm, the amount of total computation grows exponentially with the number of agents. Despite the dramatic reduction in required computation, we show that our multiagent rollout algorithm has the fundamental cost improvement property of standard rollout: it guarantees an improved performance relative to the base policy. We also discuss autonomous multiagent rollout schemes that allow the agents to make decisions autonomously through the use of precomputed signaling information, which is sufficient to maintain the cost improvement property, without any on-line coordination of control selection between the agents. For discounted and other infinite horizon problems, we also consider exact and approximate PI algorithms involving a new type of one-agent-at-a-time policy improvement operation. For one of our PI algorithms, we prove convergence to an agent-by-agent optimal policy, thus establishing a connection with the theory of teams. For another PI algorithm, which is executed over a more complex state space, we prove convergence to an optimal policy. Approximate forms of these algorithms are also given, based on the use of policy and value neural networks. These PI algorithms, in both their exact and their approximate form are strictly off-line methods, but they can be used to provide a base policy for use in an on-line multiagent rollout scheme.

An Estimation Method for Arrival Time at the Farthest Point in Case of Optimal Deployment of Heterogeneous Multi-agent Problems

An Estimation Method for Arrival Time at the Farthest Point in Case of Optimal Deployment of Heterogeneous Multi-agent Problems

DRLE: Decentralized Reinforcement Learning at the Edge for Traffic Light Control in the IoV

The Internet of Vehicles (IoV) enables real-time data exchange among vehicles and roadside units and thus provides a promising solution to alleviate traffic jams in the urban area. Meanwhile, better traffic management via efficient traffic light control can benefit the IoV as well by enabling a better communication environment and decreasing the network load. As such, IoV and efficient traffic light control can formulate a virtuous cycle. Edge computing, an emerging technology to provide low-latency computation capabilities at the edge of the network, can further improve the performance of this cycle. However, while the collected information is valuable, an efficient solution for better utilization and faster feedback has yet to be developed for edge-empowered IoV. To this end, we propose a Decentralized Reinforcement Learning at the Edge for traffic light control in the IoV (DRLE). DRLE exploits the ubiquity of the IoV to accelerate the collection of traffic data and its interpretation towards alleviating congestion and providing better traffic light control. DRLE operates within the coverage of the edge servers and uses aggregated data from neighboring edge servers to provide city-scale traffic light control. DRLE decomposes the highly complex problem of large area control. into a decentralized multi-agent problem. We prove its global optima with concrete mathematical reasoning. The proposed decentralized reinforcement learning algorithm running at each edge node adapts the traffic lights in real time. We conduct extensive evaluations and demonstrate the superiority of this approach over several state-of-the-art algorithms.

A Stochastic Characterization of the Capture Zone in Pursuit-Evasion Games

Pursuit-evasion games are used to define guidance strategies for multi-agent planning problems. Although optimal strategies exist for deterministic scenarios, in the case when information about the opponent players is imperfect, it is important to evaluate the effect of uncertainties on the estimated variables. This paper proposes a method to characterize the game space of a pursuit-evasion game under a stochastic perspective. The Mahalanobis distance is used as a metric to determine the levels of confidence in the estimation of the Zero Effort Miss across the capture zone. This information can be used to gain an insight into the guidance strategy. A simulation is carried out to provide numerical results.

Distributed Nonconvex Optimization of Multiagent Systems Using Boosting Functions to Escape Local Optima

In this article, we address the problem of multiple local optima arising due to nonconvex objective functions in cooperative multiagent optimization problems. To escape such local optima, we propose a systematic approach based on the concept of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">boosting functions</i> . The underlying idea is to temporarily transform the gradient at a local optimum into a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">boosted</i> <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">gradient</i> with a nonzero magnitude. We develop a distributed boosting scheme based on a gradient-based optimization algorithm using a novel optimal variable step size mechanism so as to guarantee convergence. Even though our motivation is based on the coverage control problem setting, our analysis applies to a broad class of multiagent problems. Simulation results are provided to compare the performance of different boosting functions families and to demonstrate the effectiveness of the boosting function approach in attaining improved (still generally local) optima.

Multi-agent reinforcement learning algorithm to solve a partially-observable multi-agent problem in disaster response

Disaster response operations typically involve multiple decision-makers, and each decision-maker needs to make its decisions given only incomplete information on the current situation. To account for these characteristics – decision making by multiple decision-makers with partial observations to achieve a shared objective –, we formulate the decision problem as a decentralized-partially observable Markov decision process (dec-POMDP) model. To tackle a well-known difficulty of optimally solving a dec-POMDP model, multi-agent reinforcement learning (MARL) has been used as a solution technique. However, typical MARL algorithms are not always effective to solve dec-POMDP models. Motivated by evidence in single-agent RL cases, we propose a MARL algorithm augmented by pretraining. Specifically, we use behavioral cloning (BC) as a means to pretrain a neural network. We verify the effectiveness of the proposed method by solving a dec-POMDP model for a decentralized selective patient admission problem. Experimental results of three disaster scenarios show that the proposed method is a viable solution approach to solving dec-POMDP problems and that augmenting MARL with BC for its pretraining seems to offer advantages over plain MARL in terms of solution quality and computation time.

GAMA: Graph Attention Multi-agent reinforcement learning algorithm for cooperation

Multi-agent reinforcement learning (MARL) is an important way to realize multi-agent cooperation. But there are still many challenges, including the scalability and the uncertainty of the environment that limit its application. In this paper, we explored to solve those problems through the graph network and the attention mechanism. Finally we succeeded in extending the existing algorithm and obtaining a new algorithm called GAMA. Specifically through the graph network, we made the environment information shared among agents. Meanwhile, the unimportant information was filtered out with the help of the attention mechanism, which helped to improve the communication efficiency. As a result, GAMA obtained the highest mean episode rewards compared to the baselines as well as excellent scalability. The reason why we choose the graph network is that understanding the relationship among agents plays a key role in solving multi-agent problems. And the graph network is very suitable for relational induction bias. Through the integration with the attention mechanism, it was shown that agents could figure out their relationship and focus on the influential environment factors in our experiment.

A Fuzzy Multi-Agent Problem: A General Depiction and its Logic Programming-based Application

A Multi-Agent Problem (MAP) may be seenas a class of planning and scheduling problems with multiple interacting intelligent agants. These problems may be naturally expressible and solveable by means of solvers of such programming languages as PROLOG.MAP is sometimes fuzzi fied because of a vagueness of information, for example – about abilities and preferences of interacting agents. Simoult aneously, MAP naturally stems from different variants of Constraints Satisfaction Problems (CSP). This property should be preserved in each fuzzy extension of this problem. According to these requirements and expectations – a Fuzzy Multi-Agent Problem (FMAP) – as referred to CSP – and its PROLOG-based solutions are considered in the paper. Finally, a brief discussion of the achieved solutions is also carried out. An effectiveness of this PROLOG-based approach exploitsa multi-valency-based approximation of fuzziness in programming contexts.

A game theoretic approach for car pricing and its energy efficiency level versus governmental sustainability goals by considering rebound effect: A case study of South Korea

This paper looks at the effects that environmental policies have on car manufacturers. A standing problem for car manufacturers is determining both a car’s price and its level of energy efficiency. The South Korean government has sustainable goals that encompass saving energy, seeking revenue, and advancing social welfare. This study highlights four decision variables in two groups for the government. The government determines the optimal value of each goal, while maintaining a threshold for the two other goals as constraints. The four decision variables are determining the car sales tax, determining the fuel tax, determining the cost of educating the public about environmental awareness, and determining the percentage of government’s cooperation in the manufacturer’s research and development expenditure. For the purpose of this study, a new function, average of vehicle kilometer traveled in a unit of time (AVKT), is suggested for calculating the average distance driven by a car over a given period of time, based on the cost of fuel, the car’s energy efficiency, the driver’s environmental awareness quotient, and the rebound effect. This newly defined multi-agent problem is solved by looking at the structure of the Stackelberg game, and equilibrium solutions are offered by implementing the backward induction procedure. By considering demand and profit for fuel, this study offers the most efficient goals and strategies for both car manufacturers and fuel producers alike. Finally, this study offers several results and insights.

Multi-agent scheduling problems under multitasking

We consider a multitasking scheduling model with multiple agents, each of which has a set of tasks to perform on a cloud manufacturing platform on a competitive basis. Each agent wishes to minimise its desirable objective function related to the completion times of its own tasks only. However, the cloud manufacturing platform wishes to minimise the objective of one agent (being long-term critical agent), while keeping the objective of each of the other agents (being short-term one-off agents) within a given limit. The objective functions considered are the maximum of a regular function (associated with each task), the total completion time, and the weighted number of late jobs. Cloud manufacturing enables multitasking scheduling, under which the processing of a selected task may be interrupted by other tasks that are available but unfinished. We ascertain the computational complexity status of each of the problems we consider and devise solution procedures, if viable, for them. We also conduct numerical studies to generate insights into the effects of multitasking on scheduling outcomes, with which the decision maker can justify making investments to adopt or avoid multitasking.

Arena: A General Evaluation Platform and Building Toolkit for Multi-Agent Intelligence

Learning agents that are not only capable of taking tests, but also innovating is becoming a hot topic in AI. One of the most promising paths towards this vision is multi-agent learning, where agents act as the environment for each other, and improving each agent means proposing new problems for others. However, existing evaluation platforms are either not compatible with multi-agent settings, or limited to a specific game. That is, there is not yet a general evaluation platform for research on multi-agent intelligence. To this end, we introduce Arena, a general evaluation platform for multi-agent intelligence with 35 games of diverse logics and representations. Furthermore, multi-agent intelligence is still at the stage where many problems remain unexplored. Therefore, we provide a building toolkit for researchers to easily invent and build novel multi-agent problems from the provided game set based on a GUI-configurable social tree and five basic multi-agent reward schemes. Finally, we provide Python implementations of five state-of-the-art deep multi-agent reinforcement learning baselines. Along with the baseline implementations, we release a set of 100 best agents/teams that we can train with different training schemes for each game, as the base for evaluating agents with population performance. As such, the research community can perform comparisons under a stable and uniform standard. All the implementations and accompanied tutorials have been open-sourced for the community at https://sites.google.com/view/arena-unity/.

A duopolistic game for designing a comprehensive energy-efficiency scheme regarding consumer features: Which energy policy is the best?

Governments design energy-efficiency programs (EEPs) to reduce the energy consumption of the energy-intensive sectors. In this paper, we propose a comprehensive EEP for the industrial and household sectors as a multi-agent problem. A sustainable supply chain consisting of an energy supplier, an energy-efficient manufacturer, and a conventional manufacturer is considered on this matter in a duopoly. Some energy policies under direct, indirect, and combined intervention are applied by the government to improve the cost-effectiveness of the program. In addition, the authors investigate the rebound effects of energy consumption in both sectors, under the consumer behavioral intentions i.e., the efficiency-related awareness, loyalty, and income level as a novel contribution. A multi-stage model is extended based on Cournot model to formulate the problem. The scenarios’ comparison and sensitivity analysis of the equilibrium decisions indicate that energy financial policies (e.g., energy tariff) encourage the manufacturers to follow the energy-efficiency path upon their high-energy-consuming products and processes. The combined intervention is more effective than the unilateral policies to reduce energy intensity and increase the energy-saving and social welfare of the industrial and household sectors, respectively. The industrial tariff also seems to be more powerful policy than consumer tariff and energy-efficiency awareness to satisfy the investigated energy-related targets. Finally, the present study provides a management point of view of a sustainable supply chain for energy policy-makers.

Simulation of football sport PID controller based on BP neural network

Multi-agent reinforcement learning in football simulation can be extended by single-agent reinforcement learning. However, compared with single agents, the learning space of multi-agents will increase dramatically with the increase in the number of agents, so the learning difficulty will also increase. Based on BP neural network as the model structure foundation, this research combines PID controller to control the process of model operation. In order to improve the calculation accuracy to improve the control effect, the prediction output is obtained through the prediction model instead of the actual measured value. In addition, with the football robot as the object, this research studies the multi-agent reinforcement learning problem and its application in the football robot. The content includes single-agent reinforcement learning, multi-agent system reinforcement learning, and ball hunting, role assignment, and action selection in football robot decision strategies based on this. The simulation results show that the method proposed in this paper has certain effects.

Multi-agent cooperation Q-learning algorithm based on constrained Markov Game

Multi-Agent system has broad application in real world, whose security performance, however, is barely considered. Reinforcement learning is one of the most important methods to resolve Multi-Agent problems. At present, certain progress has been made in applying Multi-Agent reinforcement learning to robot system, man-machine match, and automatic, etc. However, in the above area, an agent may fall into unsafe states where the agent may find it difficult to bypass obstacles, to receive information from other agents and so on. Ensuring the safety of Multi-Agent system is of great importance in the above areas where an agent may fall into dangerous states that are irreversible, causing great damage. To solve the safety problem, in this paper we introduce a Multi-Agent Cooperation Q-Learning Algorithm based on Constrained Markov Game. In this method, safety constraints are added to the set of actions, and each agent, when interacting with the environment to search for optimal values, should be restricted by the safety rules, so as to obtain an optimal policy that satisfies the security requirements. Since traditional Multi-Agent reinforcement learning algorithm is no more suitable for the proposed model in this paper, a new solution is introduced for calculating the global optimum state-action function that satisfies the safety constraints. We take advantage of the Lagrange multiplier method to determine the optimal action that can be performed in the current state based on the premise of linearizing constraint functions, under conditions that the state-action function and the constraint function are both differentiable, which not only improves the efficiency and accuracy of the algorithm, but also guarantees to obtain the global optimal solution. The experiments verify the effectiveness of the algorithm.

Damaged buildings recognition of post-earthquake high-resolution remote sensing images based on feature space and decision tree optimization

Multi-Agent system has broad application in real world, whose security performance, however, is barely considered. Reinforcement learning is one of the most important methods to resolve Multi-Agent problems. At present, certain progress has been made in applying Multi-Agent reinforcement learning to robot system, man-machine match, and automatic, etc. However, in the above area, an agent may fall into unsafe states where the agent may find it difficult to bypass obstacles, to receive information from other agents and so on. Ensuring the safety of Multi-Agent system is of great importance in the above areas where an agent may fall into dangerous states that are irreversible, causing great damage. To solve the safety problem, in this paper we introduce a Multi-Agent Cooperation Q-Learning Algorithm based on Constrained Markov Game. In this method, safety constraints are added to the set of actions, and each agent, when interacting with the environment to search for optimal values, should be restricted by the safety rules, so as to obtain an optimal policy that satisfies the security requirements. Since traditional Multi-Agent reinforcement learning algorithm is no more suitable for the proposed model in this paper, a new solution is introduced for calculating the global optimum state-action function that satisfies the safety constraints. We take advantage of the Lagrange multiplier method to determine the optimal action that can be performed in the current state based on the premise of linearizing constraint functions, under conditions that the state-action function and the constraint function are both differentiable, which not only improves the efficiency and accuracy of the algorithm, but also guarantees to obtain the global optimal solution. The experiments verify the effectiveness of the algorithm.

Learning with policy prediction in continuous state-action multi-agent decision processes

Inspired by recent attention to multi-agent reinforcement learning (MARL), the effort to provide efficient methods in this field is increasing. But, there are many issues which make this field challenging. Decision making of an agent depends on the other agents’ behavior while sharing information is not always possible. On the other hand, predicting other agents’ policies while they are also learning is a difficult task. Also, some agents in a multi-agent environment may not behave rationally. In such cases, achieving Nash equilibrium, as a target in a system with ideal behavior, is not possible and the best policy is the best response to the other agents’ policies. In addition, many real-world multi-agent problems have a continuous nature in their state and action spaces. This induces complexity in MARL scenarios. In order to overcome these challenges, we propose a new multi-agent learning method based on fuzzy least-square policy iteration. The proposed method consists of two parts: an Inner Model as one other agent policy approximation method and a multi-agent method to learn a near-optimal policy based on the others agents’ policies. Both of the proposed algorithms are applicable to problems with continuous state and action spaces. These methods can be used independently or in combination with each other. They are defined to perfectly suit each other so that the outputs of Inner Model are entirely consistent with the nature of inputs of the multi-agent method. In problems with no possibility of explicit communication, combinations of the proposed methods are recommended. In addition, theoretical analysis proves the near optimality of the policies learned by these methods. We evaluate the learning methods in problems with continuous state-action spaces: the well-known predator–prey problem and the unit commitment problem in the smart power grid. The results are satisfactory and show acceptable performance of our methods.

High order robust fast finite time containment control for multi-agent systems

This paper studies the problem of robust containment with trivial sensitivity to both initial conditions and communication topology for multi-agent systems. In this way, based on the homogeneity property of the dynamic systems, a new nonlinear high order sliding surface for containment problem is proposed. This sliding surface has a fast finite time dynamics which causes remarkably reduction of containment sensitivity to the multi-agent initial conditions and communication topology. Accordingly, a high order fast finite time containment control (HOFFT-CC) protocol is established and the containment of multiple agents to a convex hull is realized. The proposed framework solves the fast containment problem for high order dynamics that are subjected to the external disturbance and furthermore, for both directed and undirected graph topology. Moreover, because of decoupling the agents’ dynamics and converting the multi-agent problem to some single agent problems, the structure of the proposed method is simpler and more straightforward in comparison with previous works. The finite time stability of the closed loop multi-agent systems based on the homogeneity theory and Lyapunov theorem, is analyzed and proved. The proof is produced throughout the negative degree homogeneity property of the closed loop dynamics along with asymptotical stability. In addition, simulation for a general third order multi-agent system in a two-dimensional space is accomplished and the results demonstrate the trivial sensitivity of containment to both initial conditions and communication topology.