Cooperative Multi-agent Systems Research Articles

Distributed MPC for Self-Organized Cooperation of Multiagent Systems

Distributed MPC for Self-Organized Cooperation of Multiagent Systems

Sample-based bounds for coherent risk measures: Applications to policy synthesis and verification

Autonomous systems are increasingly used in highly variable and uncertain environments giving rise to the pressing need to consider risk in both the synthesis and verification of policies for these systems. This paper first develops a sample-based method to upper bound the risk measure evaluation of a random variable whose distribution is unknown. These bounds permit us to generate high-confidence verification statements for a large class of robotic systems in a sample-efficient manner. Second, we develop a sample-based method to determine solutions to non-convex optimization problems that outperform a large fraction of the decision space of possible solutions. Both sample-based approaches then permit us to rapidly synthesize risk-aware policies that are guaranteed to achieve a minimum level of system performance. To showcase our approach in simulation, we verify a cooperative multi-agent system and develop a risk-aware controller that outperforms the system's baseline controller. Our approach can be extended to account for any g-entropic risk measure.

Multi-Agent Deep Reinforcement Learning-Based Inference Task Scheduling and Offloading for Maximum Inference Accuracy under Time and Energy Constraints

The journey towards realizing real-time AI-driven IoT applications is facing a significant hurdle caused by the limited resources of IoT devices. Particularly for battery-powered edge devices, the decision between performing task inference locally or by offloading to edge servers, all while ensuring timely results and conserving energy, is a critical issue. This problem is further complicated when an edge device houses multiple local inference models. The challenge of effectively allocating inference models to tasks between local models and edge server models under strict time and energy constraints while maximizing overall accuracy is recognized as a strongly NP-hard problem and has not been addressed in the literature. Therefore, in this work we propose MASITO, a novel multi-agent deep reinforcement learning framework designed to address this intricate problem. By dividing the problem into two sub-problems namely task scheduling and edge server selection we propose a cooperative multi-agent system for addressing each sub-problem. MASITO’s design allows for faster training and more robust schedules using cooperative behavior where agents compensate for each other’s sub-optimal actions. Moreover, MASITO dynamically adapts to different network configurations which allows for high-mobility edge computing applications. Experiments on the ImageNet-mini dataset demonstrate the framework’s efficacy, outperforming genetic algorithms (GAs), simulated annealing (SA), and particle swarm optimization (PSO) in scheduling times by providing lower times ranging from 60% up to 90% while maintaining comparable average accuracy in worst-case scenarios and superior accuracy in best-case scenarios.

Minimizing Negative Side Effects in Cooperative Multi-Agent Systems using Distributed Coordination

Autonomous agents operating in real-world environments frequently encounter undesirable outcomes or negative side effects (NSEs) when working collaboratively alongside other agents. Even when agents can execute their primary task optimally when operating in isolation, their training may not account for potential negative interactions that arise in the presence of other agents. We frame the challenge of minimizing NSEs as a lexicographic decentralized Markov decision process in which we assume independence of rewards and transitions with respect to the primary assigned tasks, but recognize that addressing negative side effects creates a form of dependence among the agents. We present a lexicographic Q-learning approach to mitigate the NSEs using human feedback models while maintaining near-optimality with respect to the assigned tasks---up to some given slack. Our empirical evaluation across two domains demonstrates that our collaborative approach effectively mitigates NSEs, outperforming non-collaborative methods.

Intrinsic Action Tendency Consistency for Cooperative Multi-Agent Reinforcement Learning

Efficient collaboration in the centralized training with decentralized execution (CTDE) paradigm remains a challenge in cooperative multi-agent systems. We identify divergent action tendencies among agents as a significant obstacle to CTDE's training efficiency, requiring a large number of training samples to achieve a unified consensus on agents' policies. This divergence stems from the lack of adequate team consensus-related guidance signals during credit assignment in CTDE. To address this, we propose Intrinsic Action Tendency Consistency, a novel approach for cooperative multi-agent reinforcement learning. It integrates intrinsic rewards, obtained through an action model, into a reward-additive CTDE (RA-CTDE) framework. We formulate an action model that enables surrounding agents to predict the central agent's action tendency. Leveraging these predictions, we compute a cooperative intrinsic reward that encourages agents to align their actions with their neighbors' predictions. We establish the equivalence between RA-CTDE and CTDE through theoretical analyses, demonstrating that CTDE's training process can be achieved using N individual targets. Building on this insight, we introduce a novel method to combine intrinsic rewards and RA-CTDE. Extensive experiments on challenging tasks in SMAC, MPE, and GRF benchmarks showcase the improved performance of our method.

Adversarial Decision Making Against Intelligent Targets in Cooperative Multiagent Systems

In this paper we propose a distributed adversarial decision making approach for multi-agent system (MAS), which is supposed to detect a team of intelligent targets. The MAS is demanded to achieve the best detection benefit against the intelligent targets that can shelter part of their features and prevent the detection according to an expert defense policy. One of the key challenges is how to achieve higher detection benefit that is both determined by the target assignment action of the MAS and the non-predictable defense policy of the targets. To handle this problem, we first formulate the multi-agent decision making problem as a max-min problem of detection benefit and break it into separate parts that are easier to be optimized. Then we introduce a new variant of distributed alternating direction method of multipliers (ADMM) to search the optimal solutions under the worst defense policy that the targets choose. To overcome the lack of access to global convergence of multi-block ADMM, we add local additional variables to formulate a penalty for non-convex parts of the local objective function. The convergence to an equilibrium and the optimality of the detection benefit are empirically validated by numerical simulations. The influence of the parameter setting is also presented and can be regarded as a prior suggestion for real applications.

해상 전장 환경에서의 협력적 멀티 에이전트 강화학습 적용을 위한 시뮬레이터의 설계 및 구현

해상 전장 환경에서의 협력적 멀티 에이전트 강화학습 적용을 위한 시뮬레이터의 설계 및 구현

Robust distributed MPC for disturbed nonlinear multi-agent systems based on a mixed differential–integral event-triggered mechanism

The cooperation of multi-agent systems (MAS) based on distributed model predictive control (DMPC) is a current research hotspot, and how to more effectively handle additive disturbances and reduce resource consumption are two difficult problems that need to be solved. In view of this, this paper aims to design a more effective DMPC strategy for MAS, so as to effectively deal with the above problems without loss of the control performance. Firstly, a finite-horizon constrained optimization problem is established for MAS, in which a robustness constraint is designed to effectively handle additive disturbances. Then, a new efficient event-triggering condition is designed, which is established by considering the mixed information of differential and integral of the state error. Furthermore, based on the dual-mode control, an event-triggered robust DMPC algorithm is proposed to further reduce the resource consumption of the system. In addition, theoretical results are provided through analysis and deduction to ensure the iterative feasibility of the algorithm, stability of the closed-loop system and Zeno-free behavior. Finally, the proposed algorithm is applied to two examples for simulation and comparison to verify its effectiveness. The simulation results show that compared to DMPC and classical event-triggered DMPC methods, the proposed algorithm can reduce the average resource consumption of MAS by more than 81% and 36%, respectively, which indicates that the proposed strategy can effectively reduce the computation and communication burden of MAS without affecting the control performance.

CuDA2: An Approach for Incorporating Traitor Agents into Cooperative Multi-Agent Systems

CuDA2: An Approach for Incorporating Traitor Agents into Cooperative Multi-Agent Systems

Asynchronous Event-Triggered Distributed Predictive Control for Multiagent Systems With Parameterized Synchronization Constraints

This paper proposes an asynchronous event-triggered distributed predictive control method for the cooperation of multi-agent systems with parameterized synchronization constraints. A novel event-triggering condition related to the parameterized information received from neighboring agents is derived for each agent based on the input-to-state stability (ISS) condition. In such a framework, the distributed optimization problem is solved and the parameterized information is exchanged between neighboring agents, only when the triggering condition is satisfied. In addition, the dual mode control is adopted in this work and no information will be exchanged between the agent and its neighbors when the agent enters the invariant set, reducing the computation and communication load. Moreover, the recursive feasibility of the proposed algorithm is proved, and a sufficient condition is developed to ensure the stability of the closed-loop system. Finally, numerical simulation is given to verify the effectiveness of the proposed method.

Evolutionary optimization for risk-aware heterogeneous multi-agent path planning in uncertain environments.

Cooperative multi-agent systems make it possible to employ miniature robots in order to perform different experiments for data collection in wide open areas to physical interactions with test subjects in confined environments such as a hive. This paper proposes a new multi-agent path-planning approach to determine a set of trajectories where the agents do not collide with each other or any obstacle. The proposed algorithm leverages a risk-aware probabilistic roadmap algorithm to generate a map, employs node classification to delineate exploration regions, and incorporates a customized genetic framework to address the combinatorial optimization, with the ultimate goal of computing safe trajectories for the team. Furthermore, the proposed planning algorithm makes the agents explore all subdomains in the workspace together as a formation to allow the team to perform different tasks or collect multiple datasets for reliable localization or hazard detection. The objective function for minimization includes two major parts, the traveling distance of all the agents in the entire mission and the probability of collisions between the agents or agents with obstacles. A sampling method is used to determine the objective function considering the agents' dynamic behavior influenced by environmental disturbances and uncertainties. The algorithm's performance is evaluated for different group sizes by using a simulation environment, and two different benchmark scenarios are introduced to compare the exploration behavior. The proposed optimization method establishes stable and convergent properties regardless of the group size.

Enhancing Cooperative Multi-Agent Systems With Self-Advice and Near-Neighbor Priority Collision Control

Enhancing Cooperative Multi-Agent Systems With Self-Advice and Near-Neighbor Priority Collision Control

Model-free reinforcement learning-based energy management for plug-in electric vehicles in a cooperative multi-agent home microgrid with consideration of travel behavior

The rise in popularity of plug-in electric vehicles (PEVs) and the increasing use of renewable energy sources (RESs) have paved the way for advanced energy management systems (EMSs) that optimize energy usage and distribution in home microgrids (H-MGs). This study focuses on the integration of PEVs in H-MGs using an EMS and analyzes its impact on electrical power grids (EPGs). We examine the effect of PEV charging and discharging patterns on H-MG energy flow, considering various scenarios such as high renewable energy generation, different levels of PEV penetration, and EPG connection status. Our findings indicate that an efficient EMS can significantly enhance H-MGs’ overall efficiency by intelligently scheduling PEV charging and discharging, maximizing the use of locally generated renewable energy, and reducing peak load on the EPG. We demonstrate that a cooperative multi-agent system EMS, driven by a robust continuous and real-time fuzzy Q-learning (FQL) method, can reduce electricity market prices by 15% by increasing the use of renewable energy generation by 25%. To fully realize the benefits of EMS, we address challenges such as reducing dependence on EPG by 30%, improving battery state by 12%, and ensuring EPG stability in the face of uncertainties.

Heterogeneous multi-agent task allocation based on graph neural network ant colony optimization algorithms

Heterogeneous multi-agent task allocation is a key optimization problem widely used in fields such as drone swarms and multi-robot coordination. This paper proposes a new paradigm that innovatively combines graph neural networks and ant colony optimization algorithms to solve the assignment problem of heterogeneous multi-agents. The paper introduces an innovative Graph-based Heterogeneous Neural Network Ant Colony Optimization (GHNN-ACO) algorithm for heterogeneous multi-agent scenarios. The multi-agent system is composed of unmanned aerial vehicles, unmanned ships, and unmanned vehicles that work together to effectively respond to emergencies. This method uses graph neural networks to learn the relationship between tasks and agents, forming a graph representation, which is then integrated into ant colony optimization algorithms to guide the search process of ants. Firstly, the algorithm in this paper constructs heterogeneous graph data containing different types of agents and their relationships and uses the algorithm to classify and predict linkages for agent nodes. Secondly, the GHNN-ACO algorithm performs effectively in heterogeneous multi-agent scenarios, providing an effective solution for node classification and link prediction tasks in intelligent agent systems. Thirdly, the algorithm achieves an accuracy rate of 95.31% in assigning multiple tasks to multiple agents. It holds potential application prospects in emergency response and provides a new idea for multi-agent system cooperation.

Learning in Cooperative Multiagent Systems Using Cognitive and Machine Models

Developing effective multi-agent systems (MASs) is critical for many applications requiring collaboration and coordination with humans. Despite the rapid advance of multi-agent deep reinforcement learning (MADRL) in cooperative MASs, one of the major challenges that remain is the simultaneous learning and interaction of independent agents in dynamic environments in the presence of stochastic rewards. State-of-the-art MADRL models struggle to perform well in Coordinated Multi-agent Object Transportation Problems (CMOTPs) wherein agents must coordinate with each other and learn from stochastic rewards. In contrast, humans often learn rapidly to adapt to non-stationary environments that require coordination among people. In this article, motivated by the demonstrated ability of cognitive models based on Instance-based Learning Theory (IBLT) to capture human decisions in many dynamic decision-making tasks, we propose three variants of multi-agent IBL models (MAIBLs). The idea of these MAIBL algorithms is to combine the cognitive mechanisms of IBLT and the techniques of MADRL models to deal with coordination MASs in stochastic environments from the perspective of independent learners. We demonstrate that the MAIBL models exhibit faster learning and achieve better coordination in a dynamic CMOTP task with various settings of stochastic rewards compared to current MADRL models. We discuss the benefits of integrating cognitive insights into MADRL models.

Deep clustering of Lagrangian trajectory for multi-task learning to energy saving in intelligent buildings using cooperative multi-agent

The intelligent buildings provided various incentives to get highly inefficient energy-saving caused by the non-stationary building environments. In the presence of such dynamic excitation with higher levels of nonlinearity and coupling effect of temperature and humidity, the HVAC system transitions from underdamped to overdamped indoor conditions. This led to the promotion of highly inefficient energy use and fluctuating indoor thermal comfort. To address these concerns, this study develops a novel framework based on deep clustering of lagrangian trajectories for multi-task learning (DCLTML) and adding a pre-cooling coil in the air handling unit (AHU) to alleviate a coupling issue. The proposed DCLTML exhibits great overall control and is suitable for multi-objective optimisation based on cooperative multi-agent systems (CMAS). The framework of DCLTML is used greedy iterative training to get an optimal set of weights and tabulated as a layer for each clustering structure. Such layers can deal with the challenges of large space and its massive data. Then the layer weights of each cluster are tuned by the Quasi-Newton (QN) algorithm to make the action sequence of CMAS optimal. Such a policy of CMAS effectively manipulates the inputs of the AHU, where the agents of the AHU activate the natural ventilation and set chillers into an idle state when the outdoor temperature crosses the recommended value. So, it is reasonable to assess the impact potential of thermal mass and hybrid ventilation strategy in reducing cooling energy; accordingly, the assigning results of the proposed DCLTML show that its main cooling coil saves >40% compared to the conventional benchmarks. Besides significant energy savings and improving environmental comfort, the DCLTML exhibits superior high-speed response and robustness performance and eliminates fatigue and wear due to shuttering valves. The results show that the DCLTML algorithm is a promising new approach for controlling HVAC systems. It is more robust to environmental variations than traditional controllers, and it can learn to control the HVAC system in a way that minimises energy consumption. The DCLTML algorithm is still under development, but it can potentially revolutionise how HVAC systems are controlled.

Distributed dynamic pricing of multiple perishable products using multi-agent reinforcement learning

Revenue management (RM) is essential for a wide range of industries such as airlines, hotels, cruise lines, fashion, and seasonal retail. This paper focuses on the multi-perishable-product dynamic pricing (MPPDP) problem, a significant research field in RM, where a company sells multiple interactive and perishable products over a limited selling window without replenishment. Most studies in this field assume customer behavior, which is modeled by demand function, is known in advance. Even when considering uncertainty in customer behavior, most studies still assume the mathematical form or structural properties of the underlying demand function are known in advance. However, these assumptions are usually inconsistent with the actual market situation. Recently, Reinforcement Learning (RL), a potent technique for handling sequential decision-making problems, has been increasingly applied to solve complex dynamic pricing problems without relying on any assumption about demand functions. However, the curse of dimensionality poses a challenge for currently used centralized RL algorithms when solving the MPPDP problem due to the exponential expansion of the joint price space with the number of products. To address this issue, our paper proposes a distributed dynamic pricing framework and innovatively models the MPPDP problem as a Fully Cooperative Markov Game solved by Multi-Agent Reinforcement Learning (MARL). Additionally, we use counterfactual baselines to design appropriate agent-specific reward signals that facilitate faster learning for the agents in our established multi-agent cooperative system. Finally, two MARL-based distributed dynamic pricing algorithms, Counterfactual Q-learning, and Counterfactual DQN, are proposed for the MPPDP problem. Through the case studies on four computer-simulated markets, we show that our algorithms can alleviate the curse of dimensionality faced by centralized RL algorithms, expedite the learning process, and demonstrate satisfactory performance without relying on any assumption about demand functions. In conclusion, our work provides an effective MARL-based distributed dynamic pricing framework and algorithms for companies to efficiently price their multiple perishable products in modern highly uncertain markets.

Deep Dyna-Q for Rapid Learning and Improved Formation Achievement in Cooperative Transportation

Nowadays, academic research, disaster mitigation, industry, and transportation apply the cooperative multi-agent concept. A cooperative multi-agent system is a multi-agent system that works together to solve problems or maximise utility. The essential marks of formation control are how the multiple agents can reach the desired point while maintaining their position in the formation based on the dynamic conditions and environment. A cooperative multi-agent system closely relates to the formation change issue. It is necessary to change the arrangement of multiple agents according to the environmental conditions, such as when avoiding obstacles, applying different sizes and shapes of tracks, and moving different sizes and shapes of transport objects. Reinforcement learning is a good method to apply in a formation change environment. On the other hand, the complex formation control process requires a long learning time. This paper proposed using the Deep Dyna-Q algorithm to speed up the learning process while improving the formation achievement rate by tuning the parameters of the Deep Dyna-Q algorithm. Even though the Deep Dyna-Q algorithm has been used in many applications, it has not been applied in an actual experiment. The contribution of this paper is the application of the Deep Dyna-Q algorithm in formation control in both simulations and actual experiments. This study successfully implements the proposed method and investigates formation control in simulations and actual experiments. In the actual experiments, the Nexus robot with a robot operating system (ROS) was used. To confirm the communication between the PC and robots, camera processing, and motor controller, the velocities from the simulation were directly given to the robots. The simulations could give the same goal points as the actual experiments, so the simulation results approach the actual experimental results. The discount rate and learning rate values affected the formation change achievement rate, collision number among agents, and collisions between agents and transport objects. For learning rate comparison, DDQ (0.01) consistently outperformed DQN. DQN obtained the maximum −170 reward in about 130,000 episodes, while DDQ (0.01) could achieve this value in 58,000 episodes and achieved a maximum −160 reward. The application of an MEC (model error compensator) in the actual experiment successfully reduced the error movement of the robots so that the robots could produce the formation change appropriately.

Automatic Curriculum Learning for Large-Scale Cooperative Multiagent Systems

Recently, a lot of works have been devoted to researching how agents can learn efficient cooperation in multiagent systems. However, it still remains challenging in large-scale multiagent systems (MASs) due to the complex dynamics between the agents and environment and the dimension explosion of state-action space. In this paper, we propose a novel MultiAgent Automatic Curriculum Learning method (MA-ACL) to solve learning problems of large-scale cooperative MASs by beginning from learning on a multiagent scenario with a few agents and automatically progressively increasing the number of agents. An evaluation mechanism based on self-supervised learning is innovatively designed to automatically generate appropriate curricula with a progressively increasing number of agents. Moreover, since the observation state dimension of agents varies across curricula and the learned policy knowledge needs to be effectively encoded, we design a new Distributed Transferable Relation-modeling Policy network structure (DTRP) to handle the dynamic size of the network input and model relational knowledge between agents and their surrounding environment. Simulation results show that the proposed MA-ACL using DTRP can significantly improve the performance of large-scale multiagent learning compared with manual or non curriculum learning methods, and DTRP greatly boosts the performance of MA-ACL.

A Genetic Programming-based Framework for Semi-automated Multi-agent Systems Engineering

With the rise of new technologies, such as Edge computing, Internet of Things, Smart Cities, and Smart Grids, there is a growing need for multi-agent systems (MAS) approaches. Designing multi-agent systems is challenging, and doing this in an automated way is even more so. To address this, we propose a new framework, Evolved Gossip Contracts (EGC). It builds on Gossip Contracts (GC), a decentralised cooperation protocol that is used as the communication mechanism to facilitate self-organisation in a cooperative MAS. GC has several methods that are implemented uniquely, depending on the goal the MAS aims to achieve. The EGC framework uses evolutionary computing to search for the best implementation of these methods. To evaluate EGC, it was used to solve a classical NP-hard optimisation problem, the Bin Packing Problem (BPP). The experimental results show that EGC successfully discovered a decentralised strategy to solve the BPP, which is better than two classical heuristics on test cases similar to those on which it was trained; the improvement is statistically significant. EGC is the first framework that leverages evolutionary computing to semi-automate the discovery of a communication protocol for a MAS that has been shown to be effective at solving an NP-hard problem.