Cooperative Multi-agent Systems Research Articles

Intrinsic Action Tendency Consistency for Cooperative Multi-Agent Reinforcement Learning

Efficient collaboration in the centralized training with decentralized execution (CTDE) paradigm remains a challenge in cooperative multi-agent systems. We identify divergent action tendencies among agents as a significant obstacle to CTDE's training efficiency, requiring a large number of training samples to achieve a unified consensus on agents' policies. This divergence stems from the lack of adequate team consensus-related guidance signals during credit assignment in CTDE. To address this, we propose Intrinsic Action Tendency Consistency, a novel approach for cooperative multi-agent reinforcement learning. It integrates intrinsic rewards, obtained through an action model, into a reward-additive CTDE (RA-CTDE) framework. We formulate an action model that enables surrounding agents to predict the central agent's action tendency. Leveraging these predictions, we compute a cooperative intrinsic reward that encourages agents to align their actions with their neighbors' predictions. We establish the equivalence between RA-CTDE and CTDE through theoretical analyses, demonstrating that CTDE's training process can be achieved using N individual targets. Building on this insight, we introduce a novel method to combine intrinsic rewards and RA-CTDE. Extensive experiments on challenging tasks in SMAC, MPE, and GRF benchmarks showcase the improved performance of our method.

Adversarial Decision Making Against Intelligent Targets in Cooperative Multiagent Systems

In this paper we propose a distributed adversarial decision making approach for multi-agent system (MAS), which is supposed to detect a team of intelligent targets. The MAS is demanded to achieve the best detection benefit against the intelligent targets that can shelter part of their features and prevent the detection according to an expert defense policy. One of the key challenges is how to achieve higher detection benefit that is both determined by the target assignment action of the MAS and the non-predictable defense policy of the targets. To handle this problem, we first formulate the multi-agent decision making problem as a max-min problem of detection benefit and break it into separate parts that are easier to be optimized. Then we introduce a new variant of distributed alternating direction method of multipliers (ADMM) to search the optimal solutions under the worst defense policy that the targets choose. To overcome the lack of access to global convergence of multi-block ADMM, we add local additional variables to formulate a penalty for non-convex parts of the local objective function. The convergence to an equilibrium and the optimality of the detection benefit are empirically validated by numerical simulations. The influence of the parameter setting is also presented and can be regarded as a prior suggestion for real applications.

해상 전장 환경에서의 협력적 멀티 에이전트 강화학습 적용을 위한 시뮬레이터의 설계 및 구현

해상 전장 환경에서의 협력적 멀티 에이전트 강화학습 적용을 위한 시뮬레이터의 설계 및 구현

Robust distributed MPC for disturbed nonlinear multi-agent systems based on a mixed differential–integral event-triggered mechanism

The cooperation of multi-agent systems (MAS) based on distributed model predictive control (DMPC) is a current research hotspot, and how to more effectively handle additive disturbances and reduce resource consumption are two difficult problems that need to be solved. In view of this, this paper aims to design a more effective DMPC strategy for MAS, so as to effectively deal with the above problems without loss of the control performance. Firstly, a finite-horizon constrained optimization problem is established for MAS, in which a robustness constraint is designed to effectively handle additive disturbances. Then, a new efficient event-triggering condition is designed, which is established by considering the mixed information of differential and integral of the state error. Furthermore, based on the dual-mode control, an event-triggered robust DMPC algorithm is proposed to further reduce the resource consumption of the system. In addition, theoretical results are provided through analysis and deduction to ensure the iterative feasibility of the algorithm, stability of the closed-loop system and Zeno-free behavior. Finally, the proposed algorithm is applied to two examples for simulation and comparison to verify its effectiveness. The simulation results show that compared to DMPC and classical event-triggered DMPC methods, the proposed algorithm can reduce the average resource consumption of MAS by more than 81% and 36%, respectively, which indicates that the proposed strategy can effectively reduce the computation and communication burden of MAS without affecting the control performance.

Asynchronous Event-Triggered Distributed Predictive Control for Multiagent Systems With Parameterized Synchronization Constraints

This paper proposes an asynchronous event-triggered distributed predictive control method for the cooperation of multi-agent systems with parameterized synchronization constraints. A novel event-triggering condition related to the parameterized information received from neighboring agents is derived for each agent based on the input-to-state stability (ISS) condition. In such a framework, the distributed optimization problem is solved and the parameterized information is exchanged between neighboring agents, only when the triggering condition is satisfied. In addition, the dual mode control is adopted in this work and no information will be exchanged between the agent and its neighbors when the agent enters the invariant set, reducing the computation and communication load. Moreover, the recursive feasibility of the proposed algorithm is proved, and a sufficient condition is developed to ensure the stability of the closed-loop system. Finally, numerical simulation is given to verify the effectiveness of the proposed method.

Enhancing Cooperative Multi-Agent Systems With Self-Advice and Near-Neighbor Priority Collision Control

Enhancing Cooperative Multi-Agent Systems With Self-Advice and Near-Neighbor Priority Collision Control

Model-free reinforcement learning-based energy management for plug-in electric vehicles in a cooperative multi-agent home microgrid with consideration of travel behavior

The rise in popularity of plug-in electric vehicles (PEVs) and the increasing use of renewable energy sources (RESs) have paved the way for advanced energy management systems (EMSs) that optimize energy usage and distribution in home microgrids (H-MGs). This study focuses on the integration of PEVs in H-MGs using an EMS and analyzes its impact on electrical power grids (EPGs). We examine the effect of PEV charging and discharging patterns on H-MG energy flow, considering various scenarios such as high renewable energy generation, different levels of PEV penetration, and EPG connection status. Our findings indicate that an efficient EMS can significantly enhance H-MGs’ overall efficiency by intelligently scheduling PEV charging and discharging, maximizing the use of locally generated renewable energy, and reducing peak load on the EPG. We demonstrate that a cooperative multi-agent system EMS, driven by a robust continuous and real-time fuzzy Q-learning (FQL) method, can reduce electricity market prices by 15% by increasing the use of renewable energy generation by 25%. To fully realize the benefits of EMS, we address challenges such as reducing dependence on EPG by 30%, improving battery state by 12%, and ensuring EPG stability in the face of uncertainties.

Heterogeneous multi-agent task allocation based on graph neural network ant colony optimization algorithms

Heterogeneous multi-agent task allocation is a key optimization problem widely used in fields such as drone swarms and multi-robot coordination. This paper proposes a new paradigm that innovatively combines graph neural networks and ant colony optimization algorithms to solve the assignment problem of heterogeneous multi-agents. The paper introduces an innovative Graph-based Heterogeneous Neural Network Ant Colony Optimization (GHNN-ACO) algorithm for heterogeneous multi-agent scenarios. The multi-agent system is composed of unmanned aerial vehicles, unmanned ships, and unmanned vehicles that work together to effectively respond to emergencies. This method uses graph neural networks to learn the relationship between tasks and agents, forming a graph representation, which is then integrated into ant colony optimization algorithms to guide the search process of ants. Firstly, the algorithm in this paper constructs heterogeneous graph data containing different types of agents and their relationships and uses the algorithm to classify and predict linkages for agent nodes. Secondly, the GHNN-ACO algorithm performs effectively in heterogeneous multi-agent scenarios, providing an effective solution for node classification and link prediction tasks in intelligent agent systems. Thirdly, the algorithm achieves an accuracy rate of 95.31% in assigning multiple tasks to multiple agents. It holds potential application prospects in emergency response and provides a new idea for multi-agent system cooperation.

Learning in Cooperative Multiagent Systems Using Cognitive and Machine Models

Developing effective multi-agent systems (MASs) is critical for many applications requiring collaboration and coordination with humans. Despite the rapid advance of multi-agent deep reinforcement learning (MADRL) in cooperative MASs, one of the major challenges that remain is the simultaneous learning and interaction of independent agents in dynamic environments in the presence of stochastic rewards. State-of-the-art MADRL models struggle to perform well in Coordinated Multi-agent Object Transportation Problems (CMOTPs) wherein agents must coordinate with each other and learn from stochastic rewards. In contrast, humans often learn rapidly to adapt to non-stationary environments that require coordination among people. In this article, motivated by the demonstrated ability of cognitive models based on Instance-based Learning Theory (IBLT) to capture human decisions in many dynamic decision-making tasks, we propose three variants of multi-agent IBL models (MAIBLs). The idea of these MAIBL algorithms is to combine the cognitive mechanisms of IBLT and the techniques of MADRL models to deal with coordination MASs in stochastic environments from the perspective of independent learners. We demonstrate that the MAIBL models exhibit faster learning and achieve better coordination in a dynamic CMOTP task with various settings of stochastic rewards compared to current MADRL models. We discuss the benefits of integrating cognitive insights into MADRL models.

Deep clustering of Lagrangian trajectory for multi-task learning to energy saving in intelligent buildings using cooperative multi-agent

The intelligent buildings provided various incentives to get highly inefficient energy-saving caused by the non-stationary building environments. In the presence of such dynamic excitation with higher levels of nonlinearity and coupling effect of temperature and humidity, the HVAC system transitions from underdamped to overdamped indoor conditions. This led to the promotion of highly inefficient energy use and fluctuating indoor thermal comfort. To address these concerns, this study develops a novel framework based on deep clustering of lagrangian trajectories for multi-task learning (DCLTML) and adding a pre-cooling coil in the air handling unit (AHU) to alleviate a coupling issue. The proposed DCLTML exhibits great overall control and is suitable for multi-objective optimisation based on cooperative multi-agent systems (CMAS). The framework of DCLTML is used greedy iterative training to get an optimal set of weights and tabulated as a layer for each clustering structure. Such layers can deal with the challenges of large space and its massive data. Then the layer weights of each cluster are tuned by the Quasi-Newton (QN) algorithm to make the action sequence of CMAS optimal. Such a policy of CMAS effectively manipulates the inputs of the AHU, where the agents of the AHU activate the natural ventilation and set chillers into an idle state when the outdoor temperature crosses the recommended value. So, it is reasonable to assess the impact potential of thermal mass and hybrid ventilation strategy in reducing cooling energy; accordingly, the assigning results of the proposed DCLTML show that its main cooling coil saves >40% compared to the conventional benchmarks. Besides significant energy savings and improving environmental comfort, the DCLTML exhibits superior high-speed response and robustness performance and eliminates fatigue and wear due to shuttering valves. The results show that the DCLTML algorithm is a promising new approach for controlling HVAC systems. It is more robust to environmental variations than traditional controllers, and it can learn to control the HVAC system in a way that minimises energy consumption. The DCLTML algorithm is still under development, but it can potentially revolutionise how HVAC systems are controlled.

Distributed dynamic pricing of multiple perishable products using multi-agent reinforcement learning

Revenue management (RM) is essential for a wide range of industries such as airlines, hotels, cruise lines, fashion, and seasonal retail. This paper focuses on the multi-perishable-product dynamic pricing (MPPDP) problem, a significant research field in RM, where a company sells multiple interactive and perishable products over a limited selling window without replenishment. Most studies in this field assume customer behavior, which is modeled by demand function, is known in advance. Even when considering uncertainty in customer behavior, most studies still assume the mathematical form or structural properties of the underlying demand function are known in advance. However, these assumptions are usually inconsistent with the actual market situation. Recently, Reinforcement Learning (RL), a potent technique for handling sequential decision-making problems, has been increasingly applied to solve complex dynamic pricing problems without relying on any assumption about demand functions. However, the curse of dimensionality poses a challenge for currently used centralized RL algorithms when solving the MPPDP problem due to the exponential expansion of the joint price space with the number of products. To address this issue, our paper proposes a distributed dynamic pricing framework and innovatively models the MPPDP problem as a Fully Cooperative Markov Game solved by Multi-Agent Reinforcement Learning (MARL). Additionally, we use counterfactual baselines to design appropriate agent-specific reward signals that facilitate faster learning for the agents in our established multi-agent cooperative system. Finally, two MARL-based distributed dynamic pricing algorithms, Counterfactual Q-learning, and Counterfactual DQN, are proposed for the MPPDP problem. Through the case studies on four computer-simulated markets, we show that our algorithms can alleviate the curse of dimensionality faced by centralized RL algorithms, expedite the learning process, and demonstrate satisfactory performance without relying on any assumption about demand functions. In conclusion, our work provides an effective MARL-based distributed dynamic pricing framework and algorithms for companies to efficiently price their multiple perishable products in modern highly uncertain markets.

Deep Dyna-Q for Rapid Learning and Improved Formation Achievement in Cooperative Transportation

Nowadays, academic research, disaster mitigation, industry, and transportation apply the cooperative multi-agent concept. A cooperative multi-agent system is a multi-agent system that works together to solve problems or maximise utility. The essential marks of formation control are how the multiple agents can reach the desired point while maintaining their position in the formation based on the dynamic conditions and environment. A cooperative multi-agent system closely relates to the formation change issue. It is necessary to change the arrangement of multiple agents according to the environmental conditions, such as when avoiding obstacles, applying different sizes and shapes of tracks, and moving different sizes and shapes of transport objects. Reinforcement learning is a good method to apply in a formation change environment. On the other hand, the complex formation control process requires a long learning time. This paper proposed using the Deep Dyna-Q algorithm to speed up the learning process while improving the formation achievement rate by tuning the parameters of the Deep Dyna-Q algorithm. Even though the Deep Dyna-Q algorithm has been used in many applications, it has not been applied in an actual experiment. The contribution of this paper is the application of the Deep Dyna-Q algorithm in formation control in both simulations and actual experiments. This study successfully implements the proposed method and investigates formation control in simulations and actual experiments. In the actual experiments, the Nexus robot with a robot operating system (ROS) was used. To confirm the communication between the PC and robots, camera processing, and motor controller, the velocities from the simulation were directly given to the robots. The simulations could give the same goal points as the actual experiments, so the simulation results approach the actual experimental results. The discount rate and learning rate values affected the formation change achievement rate, collision number among agents, and collisions between agents and transport objects. For learning rate comparison, DDQ (0.01) consistently outperformed DQN. DQN obtained the maximum −170 reward in about 130,000 episodes, while DDQ (0.01) could achieve this value in 58,000 episodes and achieved a maximum −160 reward. The application of an MEC (model error compensator) in the actual experiment successfully reduced the error movement of the robots so that the robots could produce the formation change appropriately.

Automatic Curriculum Learning for Large-Scale Cooperative Multiagent Systems

Recently, a lot of works have been devoted to researching how agents can learn efficient cooperation in multiagent systems. However, it still remains challenging in large-scale multiagent systems (MASs) due to the complex dynamics between the agents and environment and the dimension explosion of state-action space. In this paper, we propose a novel MultiAgent Automatic Curriculum Learning method (MA-ACL) to solve learning problems of large-scale cooperative MASs by beginning from learning on a multiagent scenario with a few agents and automatically progressively increasing the number of agents. An evaluation mechanism based on self-supervised learning is innovatively designed to automatically generate appropriate curricula with a progressively increasing number of agents. Moreover, since the observation state dimension of agents varies across curricula and the learned policy knowledge needs to be effectively encoded, we design a new Distributed Transferable Relation-modeling Policy network structure (DTRP) to handle the dynamic size of the network input and model relational knowledge between agents and their surrounding environment. Simulation results show that the proposed MA-ACL using DTRP can significantly improve the performance of large-scale multiagent learning compared with manual or non curriculum learning methods, and DTRP greatly boosts the performance of MA-ACL.

A Genetic Programming-based Framework for Semi-automated Multi-agent Systems Engineering

With the rise of new technologies, such as Edge computing, Internet of Things, Smart Cities, and Smart Grids, there is a growing need for multi-agent systems (MAS) approaches. Designing multi-agent systems is challenging, and doing this in an automated way is even more so. To address this, we propose a new framework, Evolved Gossip Contracts (EGC). It builds on Gossip Contracts (GC), a decentralised cooperation protocol that is used as the communication mechanism to facilitate self-organisation in a cooperative MAS. GC has several methods that are implemented uniquely, depending on the goal the MAS aims to achieve. The EGC framework uses evolutionary computing to search for the best implementation of these methods. To evaluate EGC, it was used to solve a classical NP-hard optimisation problem, the Bin Packing Problem (BPP). The experimental results show that EGC successfully discovered a decentralised strategy to solve the BPP, which is better than two classical heuristics on test cases similar to those on which it was trained; the improvement is statistically significant. EGC is the first framework that leverages evolutionary computing to semi-automate the discovery of a communication protocol for a MAS that has been shown to be effective at solving an NP-hard problem.

A particle swarm inspired approach for continuous distributed constraint optimization problems

Distributed Constraint Optimization Problems (DCOPs) are a widely studied framework for coordinating interactions in cooperative multi-agent systems. In classical DCOPs, variables owned by agents are assumed to be discrete. However, in many applications, such as target tracking or sleep scheduling in sensor networks, continuous-valued variables are more suitable than discrete ones. To better model such applications, researchers have proposed Continuous DCOPs (C-DCOPs), an extension of DCOPs, that can explicitly model problems with continuous variables. The state-of-the-art approaches for solving C-DCOPs experience either onerous memory or computation overhead and are unsuitable for non-differentiable optimization problems. To address this issue, we propose a new C-DCOP algorithm, namely Particle Swarm Optimization Based C-DCOP (PCD), which is inspired by Particle Swarm Optimization (PSO), a well-known centralized population-based approach for solving continuous optimization problems. In recent years, population-based algorithms have gained significant attention in classical DCOPs due to their ability in producing high-quality solutions. Nonetheless, to the best of our knowledge, this class of algorithms has not been utilized to solve C-DCOPs and there has been no work evaluating the potential of PSO in solving classical DCOPs or C-DCOPs. In light of this observation, we adapted PSO, a centralized algorithm, to solve C-DCOPs in a decentralized manner. The resulting PCD algorithm not only produces good-quality solutions but also finds solution without any requirement for derivative calculations. Moreover, we design a crossover operator that can be used by PCD to further improve the quality of solutions found. Finally, we theoretically prove that PCD is an anytime algorithm and empirically evaluate PCD against the state-of-the-art C-DCOP algorithms in a wide variety of benchmarks.

Fuzzy Q-learning-based approach for real-time energy management of home microgrids using cooperative multi-agent system

The development of plug-in electrical vehicles (PEVs) and distributed energy resources in home microgrids (H-MGs) is gaining attention in recent years. Accordingly, it is vital to apply a suitable energy management system to provide the required energy of the PEV and ensure the health of the energy storage systems in the H-MG. This paper proposes a continuous real-time cooperative multi-agent system (MAS) for H-MG. Real-time MAS interacts with H-MG agents to perform as independent learners applying a distributed and cooperative reinforcement learning method, while state variables are shared to coordinate their real-time behavior. Therefore, the fuzzy Q-learning (FQL) method is investigated to control the agents in the continuous state space and achieve an efficient solution. The experimental results highlight the effectiveness of the suggested control strategy to guarantee the energy supply and reduce the electricity market price. The results of the simulation demonstrate that using the proposed cooperative MAS with the real-time FQL method in the energy management of the H-MG will reduce the electricity market price by 10%, increase the health of the storage system by 35.67%, and enhance the utilization of the PV system to charge PEV for 8% at the peak load demand.

Multi-agent Distributed Model Predictive Control with Connectivity Constraint

In cooperative multi-agent robotic systems, coordination is necessary in order to complete a given task. Important examples include search and rescue, operations in hazardous environments, and environmental monitoring. Coordination, in turn, requires simultaneous satisfaction of safety critical constraints, in the form of state and input constraints, and a connectivity constraint, in order to ensure that at every time instant there exists a communication path between every pair of agents in the network. In this work, we present a model predictive controller that tackles the problem of performing multi-agent coordination while simultaneously satisfying safety critical and connectivity constraints. The former is formulated in the form of state and input constraints and the latter as a constraint on the second smallest eigenvalue of the associated communication graph Laplacian matrix, also known as Fiedler eigenvalue, which enforces the connectivity of the communication network. We propose a sequential quadratic programming formulation to solve the associated optimization problem that is amenable to distributed optimization, making the proposed solution suitable for control of multi-agent robotics systems relying on local computation. Finally, the effectiveness of the algorithm is highlighted with a numerical simulation.

Distributed Model Predictive Control for Periodic Cooperation of Multi-Agent Systems

We consider multi-agent systems with heterogeneous, nonlinear agents subject to individual constraints that want to achieve a periodic, dynamic cooperative control goal which can be characterised by a set and a suitable cost. We propose a sequential distributed model predictive control (MPC) scheme in which agents sequentially solve an individual optimisation problem to track an artificial periodic output trajectory. The optimisation problems are coupled through these artificial periodic output trajectories, which are communicated and penalised using the cost that characterises the cooperative goal. The agents communicate only their artificial trajectories and only once per time step. We show that under suitable assumptions, the agents can incrementally move their artificial output trajectories towards the cooperative goal, and, hence, their closed-loop output trajectories asymptotically achieve it. We illustrate the scheme with a simulation example.

Zeroth-Order Non-Convex Optimization for Cooperative Multi-Agent Systems with Diminishing Step Size and Smoothing Radius

We study a class of zeroth-order distributed optimization problems, where each agent can control a partial vector and observe a local cost that depends on the joint vector of all agents, and the agents can communicate with each other with time delay. We propose and study a gradient descent-based algorithm using two-point gradient estimators with diminishing smoothing parameters and diminishing step-size and we establish the convergence rate to a first-order stationary point for general nonconvex problems. A byproduct of our proposed method with diminishing step size and smoothing parameters, as opposed to the fixed-parameter scheme, is that our proposed algorithm does not require any information regarding the local cost functions. This makes the solution appealing in practice as it allows for optimizing an unknown (black-box) global function without prior knowledge of its smoothness parameters. At the same time, the performance will adaptively match the problem instance parameters.

Towards a more efficient computation of individual attribute and policy contribution for post-hoc explanation of cooperative multi-agent systems using Myerson values

A quantitative assessment of the global importance of an agent in a team is as valuable as gold for strategists, decision-makers, and sports coaches. Yet, retrieving this information is not trivial since in a cooperative task it is hard to isolate the performance of an individual from the one of the whole team. Moreover, it is not always clear the relationship between the role of an agent and his personal attributes. In this work we conceive an application of the Shapley analysis for studying the contribution of both agent policies and attributes, putting them on equal footing. Since the computational complexity is NP-hard and scales exponentially with the number of participants in a transferable utility coalitional game, we resort to exploiting a-priori knowledge about the rules of the game to constrain the relations between the participants over a graph. We hence propose a method to determine a Hierarchical Knowledge Graph of agents’ policies and features in a Multi-Agent System. Assuming a simulator of the system is available, the graph structure allows to exploit dynamic programming to assess the importances in a much faster way. We test the proposed approach in a proof-of-case environment deploying both hardcoded policies and policies obtained via Deep Reinforcement Learning. The proposed paradigm is less computationally demanding than trivially computing the Shapley values and provides great insight not only into the importance of an agent in a team but also into the attributes needed to deploy the policy at its best.