Multi-agent Learning System Research Articles

Mathematics of multi-agent learning systems at the interface of game theory and artificial intelligence

Mathematics of multi-agent learning systems at the interface of game theory and artificial intelligence

A Multiagent Cooperative Learning System With Evolution of Social Roles

Recent developments in reinforcement learning have been able to derive optimal policies for sophisticated and capable agents, and shown to achieve human-level performance on a number of challenging tasks. Unfortunately, when it comes to multi-agent systems (MASs), complexities such as non-stationarity and partial observability bring new challenges to the field. Building a flexible and efficient multi-agent reinforcement learning algorithm capable of handling complex tasks has to date remained an open challenge. This paper presents an Multi-Agent learning system with the evolution of Social Roles (eSRMA). The main interest is placed on solving the key issues in the definition and evolution of suitable roles, and optimizing the policies accompanied by social roles in MAS efficiently. Specifically, eSRMA incorporates and cultivates role division awareness of agents to improve the ability to deal with complex cooperative tasks. Each agent is assigned a role module, which can dynamically generate roles based on the individuals’ local observations. A novel multi-agent reinforcement learning algorithm (MARL) is designed as the principal driving force that governs the role-policy learning process by a role-attention credit assignment mechanism. Moreover, a role evolution process is developed to help agents dynamically choose appropriate roles in decision-making. Comprehensive experiments on the StarCraft II micromanagement benchmark have demonstrated that eSRMA exhibits superiority in achieving higher learning capability and efficiency for multiple agents compared to the state-of-the-art MARL methods.

RevAP: A bankruptcy-based algorithm to solve the multi-agent credit assignment problem in task start threshold-based multi-agent systems

Multi-Agent Systems (MASs) are the prominent symbol of Distributed Artificial Intelligence (DAI). Learning in MAS, which is commonly based on Reinforcement Learning (RL), is one of the problems that play an essential role in unknown environments. In this study aimed at solving the Multi-agent Credit Assignment (MCA) problem, we introduce the Task Start Threshold (TST) of agents as a new constraint in a multi-score operational environment, transforming the MCA into a bankruptcy problem. In the following, considering the bankruptcy concept, a new base algorithm, which is called Reverse Adjusted Proportional (RevAP), is introduced. Based on this algorithm, three methods PTST, T-MAS, and T-KAg, were presented to solve the MCA with different strategies. The proposed methods were evaluated in terms of group learning rate, confidence, expertness, certainty, efficiency, correctness, and density in comparison to the state-of-the-art methods such as ranking methods, dynamic, history-based as knowledge-based methods, Counterfactual Multi-Agent Policy Gradient (COMA) as an example of policy-based methods, Value-Decomposition Network (VDN) as an example of value-based methods, and Shapley Q- value Deep Deterministic Policy Gradient (SQDDPG) as a game theory-based method. The results reveal the better performance of the proposed approach compared to the existing methods based on the majority of the parameters.

Prosocial dynamics in multiagent systems

AbstractMeeting today's major scientific and societal challenges requires understanding dynamics of prosociality in complex adaptive systems. Artificial intelligence (AI) is intimately connected with these challenges, both as an application domain and as a source of new computational techniques: On the one hand, AI suggests new algorithmic recommendations and interaction paradigms, offering novel possibilities to engineer cooperation and alleviate conflict in multiagent (hybrid) systems; on the other hand, new learning algorithms provide improved techniques to simulate sophisticated agents and increasingly realistic environments. In various settings, prosocial actions are socially desirable yet individually costly, thereby introducing a social dilemma of cooperation. How can AI enable cooperation in such domains? How to understand long‐term dynamics in adaptive populations subject to such cooperation dilemmas? How to design cooperation incentives in multiagent learning systems? These are questions that I have been exploring and that I discussed during the New Faculty Highlights program at AAAI 2023. This paper summarizes and extends that talk.

Inferring Intents From Equivariant–Invariant Representations and Relational Learning in Multiagent Systems

Inferring Intents From Equivariant–Invariant Representations and Relational Learning in Multiagent Systems

Cogni-Sec: A secure cognitive enabled distributed reinforcement learning model for medical cyber–physical system

The advent of the Internet of Things (IoT) has resulted in significant technical development in the healthcare sector, enabling the establishment of Medical Cyber–Physical Systems (MCPS). The increased number of MCPS generates a massive amount of privacy-sensitive data, hence it is important to enhance the security of devices and data transmission in MCPS. Earlier several research studies were undertaken in order to enhance security in healthcare, but none of them could adapt to changing behaviors of data attacks. Here the role of blockchain and Reinforcement Learning (RL) comes into play since it can adjust itself to the nature of changing attacks, thus preventing any kind of attacks. This work proposes a solution, named Cogni-Sec, which employs a decentralized cognitive blockchain and Reinforcement Learning architecture and addresses the security issue. Blockchain is incorporated in the approach for data storage and transmission to increase the degree of security in the MCPS modules. Hyperledger Fabric is applied as the blockchain base which shows transaction query results with nearly 10% increased throughput, 69% less memory consumption, and 15% lower CPU usage when compared to Ethereum. Further security risk at the block mining level within a blockchain network is reduced by introducing distributed Reinforcement Learning architecture in replacement for the miner nodes, which imitates the cognitive behavior of miners in a distributed environment. Different multi-agent learning systems have been evaluated for building the mining agent. Among these, the a3c agent in distributed learning setup yields the optimum cumulative reward with a median value of 54.5 and minimizes the maximum number of data threats.

Adaptive-Modeling Multi-Agent Learning System for Video Behavioral Clustering Recognition

Multi-agent systems are suitable for handling complex problems due to their high parallelism and autonomous evolution ability. In this paper, we propose an adaptive clustering multi-agent learning system for intelligent applications with continuously changing requirements. Each agent model changes temporal sequences using the longest common subsequence (LCS) algorithm. Multiple agents collaborate in a multilayer decentralized approach to enhance learning adaptability and achieve self-supervised behavioral clustering. The system is constructed using a “memory-like” method and operates primarily on memory access and comparison, avoiding extensive matrix operations of artificial neural networks while achieving learning and prediction functions. We chose an unsupervised vehicle behavioral clustering scenario for feasibility validation in which the system’s cognitive objective is to cluster and recognize vehicle behaviors. In a low computational environment, the system can complete clustering functions and exhibit continuous learning capabilities when new behavioral changes occur. The proposed approach achieves an accuracy of 97.4% while processing at a speed 1–5 times faster than similar clustering algorithms. The verification results indicate that this system has excellent potential to enhance intelligent sensing front ends.

Use of Social Interaction and Intention to Improve Motion Prediction Within Automated Vehicle Framework: A Review

Human errors contribute to 94%(±2.2%) of road crashes resulting in fatal/non-fatal causalities, vehicle damages and a predicament in the pathway to safer road systems. Automated Vehicles (AVs) have been a potential attempt in lowering the crash rate by replacing human drivers with an advanced computer-aided decision-making approach. However, AVs are yet to progress in handling the unprecedented situations involving interactions with other road users. This raises a need for a sophisticated and robust methodological framework to predict human driver interaction and intention. It is of prime importance to develop a constructive knowledge on the existing literature for a proficient forward leap in the field. To address this, we aim to conduct a comprehensive review on motion prediction methods in automated driving context with a special emphasis on model-based and data-driven approaches. Over a hundred studies related to the motion prediction for AVs have been extensively reviewed. This study recommends that the field requires more intricate classification of motion prediction methods, as the conventional three-level categorisation scheme should be upgraded to a profound and present-day context. Therefore, we attempt to provide a clear categorisation of existing motion prediction solutions by adopting four principal strategies: 1. Prediction methods, 2. Classes, 3. Algorithms and 4. Datasets. An all-inclusive summary of the reviewed studies with their respective pros and cons are also presented. Furthermore, we summarise the standard evaluation metrics applied for road users’ intention estimation and trajectory prediction tasks. It is found that the recent studies are built upon multi-agent learning systems with interaction among multiple road users in the same road environment. These methods can provide reliable prediction performance in highly interactive situations over long periods of time. However, the limitation could be at the cost of higher computational complexity in comparison to conventional methods, which are simpler to design and computationally effective. It is also observed that the conventional methods can only operate over a narrow prediction horizon and seldom consider the interactions among the road users. This review contributes to knowledge in validation, addresses the discrepancies, to explicate the ambiguities and to streamline current research for a futuristic perspective beneficiary in motion prediction field.

Behavior Reasoning for Opponent Agents in Multi-Agent Learning Systems

One important component of developing autonomous agents lies in the accurate prediction of their opponents’ behaviors when the agents interact with others in an uncertain environment. Most recent study focuses on first constructing predictive types (or models) of the opponents, considering their various properties of interest, and subsequently using these models to predict their behaviors accordingly. However, as the possible type space can be rather large, it is time-consuming, and sometimes even infeasible, to predict the actual behaviors of opponents with all candidate types. Thus, in this paper a tractable opponent behavior reasoning approach is proposed that facilitates ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$a$</tex-math></inline-formula> ) extraction of a small yet representative summary of all candidates using sub-modular-type maximization, and accordingly, ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$b$</tex-math></inline-formula> ) identification of the most appropriate type for real-time behavior prediction based on multi-armed bandits. In addition, we propose a knowledge-transfer scheme through demonstration learning to synchronize subject agents’ knowledge about their opponents’ behaviors. This further reduces the burden of reasoning with all models of their opponents from the perspective of individual subject agents. We integrate the new behavior prediction and reasoning method into a state-of-the-art evolutionary multi-agent framework, namely a memetic multi-agent system (MeMAS), and demonstrate empirical performance in two problem domains.

From intelligent agents to trustworthy human-centred multiagent systems

The Agents, Interaction and Complexity research group at the University of Southampton has a long track record of research in multiagent systems (MAS). We have made substantial scientific contributions across learning in MAS, game-theoretic techniques for coordinating agent systems, and formal methods for representation and reasoning. We highlight key results achieved by the group and elaborate on recent work and open research challenges in developing trustworthy autonomous systems and deploying human-centred AI systems that aim to support societal good.

Risk Adversarial Learning System for Connected and Autonomous Vehicle Charging

In this article, the design of a rational decision support system (RDSS) for a connected and autonomous vehicle charging infrastructure (CAV-CI) is studied. In the considered CAV-CI, the distribution system operator (DSO) deploys electric vehicle supply equipment (EVSE) to provide an electrical vehicle (EV) charging facility for human-driven connected vehicles (CVs) and AVs. The charging request by the human-driven EV becomes irrational when it demands more energy and charging period than its actual need. Therefore, the scheduling policy of each EVSE must be adaptively accumulated the irrational charging request to satisfy the charging demand of both CVs and autonomous vehicles (AVs). To tackle this, we formulate an RDSS problem for the DSO, where the objective is to maximize the charging capacity utilization by satisfying the laxity risk of the DSO. Thus, we devise a rational reward maximization problem to adapt the irrational behavior by CVs in a data-informed manner. We propose a novel risk adversarial multiagent learning system (RAMALS) for CAV-CI to solve the formulated RDSS problem. In RAMALS, the DSO acts as a centralized risk adversarial agent (RAA) for informing the laxity risk to each EVSE. Subsequently, each EVSE plays the role of a self-learner agent to adaptively schedule its own EV sessions by coping advice from RAA. The experiment results show that the proposed RAMALS affords around 46.6% improvement in charging rate, about 28.6% improvement in the EVSE’s active charging time, and at least 33.3% more energy utilization, as compared to a currently deployed ACN EVSE system, and other baselines.

Blockchain integrated multi-agent system for breast cancer diagnosis

&lt;span&gt;The integration of multi-agent system and blockchain technology can be beneficial to healthcare applications by providing intelligent data analysis with security. This paper presents an architecture that integrates multi-agent learning system and blockchain technology to support breast cancer diagnosis in a secured manner. The proposed system is based on a parallel hybrid fuzzy logic approach for supporting the prediction of breast cancer disease. The proposed system showed a classification accuracy of 96.49% in breast cancer diagnosis when testing with the Wisconsin Diagnostic Breast Cancer dataset. The blockchain is used to provide agent security in the proposed system to ensure that the only trusted and reputed agents are participated in the decision-making process.&lt;/span&gt;

Exploration-exploitation in multi-agent learning: Catastrophe theory meets game theory

Exploration-exploitation is a powerful and practical tool in multi-agent learning (MAL); however, its effects are far from understood. To make progress in this direction, we study a smooth analogue of Q-learning. We start by showing that our learning model has strong theoretical justification as an optimal model for studying exploration-exploitation. Specifically, we prove (1) that smooth Q-learning has bounded regret in arbitrary games for a cost model that explicitly balances game-rewards and exploration-costs, i.e., costs from testing potentially suboptimal actions, and (2) that it always converges to the set of quantal-response equilibria (QRE), the standard solution concept for games with bounded rationality, in arbitrary weighted potential games. In our main task, we then turn to measure the effect of exploration on collective system performance. We characterize the geometry of the QRE surface in low-dimensional MAL systems and link our findings with catastrophe (bifurcation) theory. In particular, as the exploration hyperparameter evolves over-time, the system undergoes phase transitions where the number and stability of equilibria can change radically given an infinitesimal change to the exploration parameter. Based on this, we provide a formal theoretical treatment of how tuning the exploration parameter can provably lead to equilibrium selection with both positive as well as negative (and potentially unbounded) effects to system performance.

Experience Weighted Learning in Multiagent Systems

Agents face challenges to achieve adaptability and stability when interacting with dynamic counterparts in a complex multiagent system (MAS). To strike a balance between these two goals, this paper proposes a learning algorithm for heterogeneous agents with bounded rationality. It integrates reinforcement learning as well as fictitious play to evaluate the historical information and adopt mechanisms in evolutionary game to adapt to uncertainty, which is referred to as experience weighted learning (EWL) in this paper. We have conducted multiagent simulations to test the performance of EWL in various games. The results demonstrate that the average payoff of EWL exceeds that of the baseline in all 4 games. In addition, we find that most of the EWL agents converge to pure strategy and become stable finally. Furthermore, we test the impact of 2 import parameters, respectively. The results show that the performance of EWL is quite stable and there is a potential to improve its performance by parameter optimization.

Exploration-Exploitation in Multi-Agent Learning: Catastrophe Theory Meets Game Theory

Exploration-exploitation is a powerful and practical tool in multi-agent learning (MAL), however, its effects are far from understood. To make progress in this direction, we study a smooth analogue of Q-learning. We start by showing that our learning model has strong theoretical justification as an optimal model for studying exploration-exploitation. Specifically, we prove that smooth Q-learning has bounded regret in arbitrary games for a cost model that explicitly captures the balance between game and exploration costs and that it always converges to the set of quantal-response equilibria (QRE), the standard solution concept for games under bounded rationality, in weighted potential games with heterogeneous learning agents. In our main task, we then turn to measure the effect of exploration in collective system performance. We characterize the geometry of the QRE surface in low-dimensional MAL systems and link our findings with catastrophe (bifurcation) theory. In particular, as the exploration hyperparameter evolves over-time, the system undergoes phase transitions where the number and stability of equilibria can change radically given an infinitesimal change to the exploration parameter. Based on this, we provide a formal theoretical treatment of how tuning the exploration parameter can provably lead to equilibrium selection with both positive as well as negative (and potentially unbounded) effects to system performance.

Formal Reachability Analysis for Multi-Agent Reinforcement Learning Systems

Reachability analysis is one of the most basic and challenging problems in verification. We investigate this problem in multi-agent reinforcement learning (MARL) system by transforming the reachability analysis to decision-making problem tackled by mixed integer linear programming (MILP) solver Gurobi. We define syntax and semantics for multi-role strategy logic (mrSL) which is used to describe the reachability specification. The logic mrSL is a variant of SL to express properties, which provide a holistic perspective to describe reachability properties instead of specifying a specific agent reaches a certain state in the MARL system. And the algorithms to translate reachability property into MILP constraints are provided. A tool called MAReachAnalysis is introduced, which uses Gurobi to solve the corresponding reachability problem and evaluate it on the predator-prey task of multi-agent deep deterministic policy gradient (MADDPG) example, discussion of the experimental results obtained on a range of test cases.

Decentralized Function Approximated Q-Learning in Multi-Robot Systems For Predator Avoidance

The nature-inspired behavior of collective motion is found to be an optimal solution in swarming systems for predator avoidance and survival. In this work, we propose a two-level control architecture for multi-robot systems (MRS), which leverages the advantages of flocking control and function approximated reinforcement learning for predator avoidance task. Reinforcement learning in multi-agent systems has gained a tremendous amount of interest in recent few years. Computationally intensive architectures such as deep reinforcement learning and actor-critic approaches have been extensively developed and have proved to be extremely efficient. The proposed approach, comprising of cooperative function approximated Q-learning, is applied such that it ensures formation maintenance in MRS while predator avoidance. A consensus filter is incorporated into the control architecture, to sense predators in close vicinity in a distributed and cooperative fashion to ensure consensus on states among the robots in the system. The proposed approach is proved to be convergent and results in superior performance in unexplored states and reduced number of variables. Simulation results confirm the effectiveness of the proposed approach over existing methods. We expect that the proposed approach can be conveniently applied to many other areas with little modifications, such as fire-fighting robots, surveillance and patrolling robots.

Implementation of an Adaptive Learning System based on Agents and Web Services

Today, the integration of web services and agent technology into Internet applications has attracted the attention of many researchers, so that these applications allow a web service to call an agent service and vice versa. Web services are emerging and promising technologies for the development, deploy-ment and integration of the Internet applications and the use of agents makes them dynamic and automatic, they can provide updates when there is new infor-mation available and improve the qualities of web services by exploiting the ca-pacities and the characteristics of agents. In this context, we propose a prototype of a multi-agent adaptive learning system based on Incremental Hybrid Case Based Reasoning in order to support the learner in his learning process by offer-ing him a learning path adapted to his profile and predict his future learning. This support will be achieved through the execution of a hybrid cycle of Case Based Reasoning which brings together a set of agents collaborating and interacting with each other to provide specific services.

Learning in multi-agent systems with asymmetric information structure

In this paper, we study multi-agent systems with asymmetric information structure. Due to limited channel capacity in communication network, the information in routing path suffers a transmission delay. Instead of the game theoretic setting, we formulate the problem as an online quadratic optimization problem subject to stochastic systems involving input delay. Since the probability statistics of system noise is unknown, the decision-maker can not utilize the traditional optimal control strategies. Motivated by online convex optimization theory, we introduce the notion of regret, which measures the cumulative performance difference between the optimal statistics known (offline) index value and the statistics unknown (online) index value. The contributions of this paper are twofold. First, utilizing the linear minimum mean square biased estimate, we derive a learning based control policy and then characterize its behavior. Second, under some basic assumptions, we further prove that the regret grows at a sub-linear rate and it is explicitly bounded by O(lnT).

Distributed No-Regret Learning in Multiagent Systems: Challenges and Recent Developments

Game theory is a well-established tool for studying interactions among self-interested players. Under the assumption of complete information on the game composition at each player, the focal point of game-theoretic studies has been on the Nash equilibrium (NE) in analyzing game outcomes and predicting strategic behaviors of rational players.