Multi-agent Learning Research Articles

A Strategy Adaptive Evolution Approach Based on the Public Goods Game

Cluster behavior is prevalent in nature. Many individuals change their behavior to adapt to a dynamically changing environment by following simple rules of behavior and interacting with information from neighboring individuals. In this study, the traditional public goods game model is improved by combining the advantages of game theory and interactive learning. A strategy adaptive evolution method based on a public goods game is proposed. The emergence of cooperative behavior in weighted networks under the co-evolution of game strategies and node weights is explored in conjunction with multi-agent interactive learning. The results show that in a public goods game with strategic adaptation, a person’s influence becomes greater if their level of adaptation exceeds the desired level, and less otherwise. This weight adjustment is defined by the intensity parameter δ. A moderate δ value can effectively facilitate the occurrence of cooperative evolution. The level of cooperation depends mainly on the weight distribution of participants, which leads to the formation of cooperative clusters controlled by high-weighted cooperators. Even with the great temptation to defect, these cooperators can prevail over defectors. The adjustment of node weights increases the heterogeneity of individuals. This research provides a viable pathway to solve social dilemmas and will further promote the application of multi-agent intelligent decision making.

A Study on Confidence: An Unsupervised Multiagent Machine Learning Experiment

A Study on Confidence: An Unsupervised Multiagent Machine Learning Experiment

Reward-Reinforced Generative Adversarial Networks for Multi-Agent Systems

Multi-agent systems deliver highly resilient and adaptable solutions for common problems in telecommunications, aerospace, and industrial robotics. However, achieving an optimal global goal remains a persistent obstacle for collaborative multi-agent systems, where learning affects the behaviour of more than one agent. A number of nonlinear function approximation methods have been proposed for solving the Bellman equation, which describe a recursive format of an optimal policy. However, how to leverage the value distribution based on reinforcement learning, and how to improve the efficiency and efficacy of such systems remain a challenge. In this work, we developed a reward-reinforced generative adversarial network to represent the distribution of the value function, replacing the approximation of Bellman updates. We demonstrated our method is resilient and outperforms other conventional reinforcement learning methods. This method is also applied to a practical case study: maximising the number of user connections to autonomous airborne base stations in a mobile communication network. Our method maximises the data likelihood using a cost function under which agents have optimal learned behaviours. This reward-reinforced generative adversarial network can be used as a generic framework for multi-agent learning at the system level.

Cooperative Cache in Cognitive Radio Networks: A Heterogeneous Multi-Agent Learning Approach

Deploying distributed cache in cognitive radio networks (CRNs), which spreads popular contents to the edge of network during the off-peak time through spectrum sharing, can reduce the deliver delay to users nearby without causing severe interference to the primary network. However, due to the un-predicable contents requirement as well as the band occupation of primary users, it is non-trivial to optimize the cache storage and contents fetching strategy of users dynamically. The letter proposes a heterogeneous multi-agent deep deterministic policy gradient (MADDPG) approach, which takes users and cache servers as two different types of agents to learn the cooperation and competition for mutual benefits. The numeral simulation demonstrates that comparing with the other single or homogeneous deep reinforcement learning (DRL) approaches, the proposed heterogeneous MADDPG can further reduce the delivery delay of users and enhance the cache efficiency of SBSs.

Blockchain integrated multi-agent system for breast cancer diagnosis

<span>The integration of multi-agent system and blockchain technology can be beneficial to healthcare applications by providing intelligent data analysis with security. This paper presents an architecture that integrates multi-agent learning system and blockchain technology to support breast cancer diagnosis in a secured manner. The proposed system is based on a parallel hybrid fuzzy logic approach for supporting the prediction of breast cancer disease. The proposed system showed a classification accuracy of 96.49% in breast cancer diagnosis when testing with the Wisconsin Diagnostic Breast Cancer dataset. The blockchain is used to provide agent security in the proposed system to ensure that the only trusted and reputed agents are participated in the decision-making process.</span>

Dynamic Coordination-Based Reinforcement Learning for Driving Policy

With the development of communication technology and artificial intelligence technology, intelligent vehicle has become a very important part of Internet of Things technology. At present, the single-vehicle intelligence is gradually improved, and more and more unmanned vehicles appear on the road. In the future, these individual intelligence applications need to be transformed into collective intelligence to give full play to the greater advantages of unmanned driving. For example, individual intelligence is self-interest. If there is no collective cooperation, it may affect the whole traffic flow for its own speed. Although the vehicle ad hoc network technology provides a guarantee for the communication between vehicles and makes cooperation between vehicles possible, there are still challenges in how to adapt to coordination learning. Coordination reinforcement learning is one of the most promising methods to solve the multiagent coordination optimization problems. However, existing coordinative learning approaches that usually rely on static topologies cannot be easily adopted to solve the vehicle coordination problems in the dynamic environment. We propose a dynamic coordination reinforcement learning to help vehicles make their driving decisions. First, we apply driving safety field theory to construct the dynamic coordination graph (DCG), representing the dynamic coordination behaviors among vehicles. Second, we design reinforcement learning techniques on our DCG model to implement the joint optimal action reasoning for the multivehicle system and eventually derive the optimal driving policy for each vehicle. Finally, compared with other multiagent learning methods, our method has a significant improvement in security and speed, which is about 1% higher than other multiagent learning methods, but its training speed is also significantly improved about 8%.

Distortion based potential game for distributed coverage control

In this work, we propose a multi-agent learning framework to address the mobile sensor coverage problem in which the minimum mutual information between the agents (mobile sensors) and their environment defines the agent-strategy selection rule towards the system Nash equilibrium in a potential game setting. Initially, the agents infer the environment behavior by means of the Gaussian process regression (GPR) approach, using their own information and the information provided by their neighborhood. Then, the rate distortion function (RDF) is used to minimize the mutual information between each agent and its environment by means of the Blahut-Arimoto algorithm, from which, the resulting conditional probability and the parameter λ have the highest relevance, since, the former describes the similitude between the agent information and its environment, and the latter, due to its influence in the distortion and gathered environment information, defines the rationality measure in the learning process. Finally, the expected distortion defined in the RDF, allows the formulation of a distortion based potential function and the consequent equilibrium convergence.

Scalable multi-agent reinforcement learning for distributed control of residential energy flexibility

This paper proposes a novel scalable type of multi-agent reinforcement learning-based coordination for distributed residential energy. Cooperating agents learn to control the flexibility offered by electric vehicles, space heating and flexible loads in a partially observable stochastic environment. In the standard independent Q-learning approach, the coordination performance of agents under partial observability drops at scale in stochastic environments. Here, the novel combination of learning from off-line convex optimisations on historical data and isolating marginal contributions to total rewards in reward signals increases stability and performance at scale. Using fixed-size Q-tables, prosumers are able to assess their marginal impact on total system objectives without sharing personal data either with each other or with a central coordinator. Case studies are used to assess the fitness of different combinations of exploration sources, reward definitions, and multi-agent learning frameworks. It is demonstrated that the proposed strategies create value at individual and system levels thanks to reductions in the costs of energy imports, losses, distribution network congestion, battery depreciation and greenhouse gas emissions.

Multi-agent learning algorithms for content placement in cache-enabled small cell networks: 4G and 5G use cases

Multi-agent learning algorithms for content placement in cache-enabled small cell networks: 4G and 5G use cases

Reinforcement learning-driven local transactive energy market for distributed energy resources

Local energy markets are emerging as a tool for coordinating generation, storage, and consumption of energy from distributed resources. In combination with automation, they promise to provide an effective energy management framework that is fair and brings system-level savings. The cooperative–competitive nature of energy markets calls for multi-agent based automation with learning energy trading agents. However, depending on the dynamics of the agent–environment interaction, this approach may yield unintended behavior of market participants. Thus, the design of market mechanisms suitable for reinforcement learning agents must take into account this interplay. This article introduces autonomous local energy exchange (ALEX) as an experimental framework that combines multi-agent learning and double auction mechanism. Participants determine their internal price signals and make energy management decisions through market interactions, rather than relying on predetermined external price signals. The main contribution of this article is examination of compatibility between specific market elements and independent learning agents. Effects of different market properties are evaluated through simulation experiments, and the results are used for determine a suitable market design. The results show that market truthfulness maintains demand-response functionality, while weak budget balancing provides a strong reinforcement signal for the learning agents. The resulting agent behavior is compared with two baselines: net billing and time-of-use rates. The ALEX-based pricing is more responsive to fluctuations in the community net load compared to the time-of-use. The more accurate accounting of renewable energy usage reduced bills by a median 38.8% compared to net billing, confirming the ability to better facilitate demand response.

Online parking assignment in an environment of partially connected vehicles: A multi-agent deep reinforcement learning approach

The advent of connected vehicles (CVs) provides new opportunities to address urban parking issues due to the widespread application of online parking assignment (OPA) services. However, before CVs fully replace non-connected vehicles (NCVs), it is envisioned that CVs and NCVs will coexist for a long time. This brings challenges for OPA because of the availability constraints imposed by the uncertain arrivals and departures of NCVs. This paper proposes a multi-agent deep reinforcement learning framework to generate efficient OPA strategies with partial observations of parking demand. Specifically, we create two agents, one for measuring the impact of NCVs and the other for exploring the parking characteristics of CVs. A value decomposition method is adopted to solve the multi-agent learning problem, and a modified exploration strategy is designed to direct agent training and avoid unnecessary trials. To verify the performance of the proposed approach, we derive the baselines of the total time expenditure in a parking area based on the widely adopted first-come-first-served strategy and a hypothetical system optimum strategy, respectively. Also, we present a dynamic assignment model with forecasting as a comparison of the proposed approach with the same demand information. Two typical parking scenarios are selected to conduct comparative experiments with actual operating data. The experimental results show that the proposed learning-based approach can effectively allocate parking resources. Provided with user parking information of CVs short in advance, our approach can achieve up to 15% improvement in assignment performance compared with other baselines.

Multi-agent learning mechanism design and simulation of multi-echelon supply chain

A supply chain multi-agent learning mechanism based on the Particle Swarm Optimization algorithm is designed. The manufacturer’s profit and the product utility are taken as objective functions to explore the influences of social learning ability and self-learning ability on the manufacturer’s optimal price, optimal advertising investment, optimal profit, product utility, and supply chain competition. This research demonstrates the influences of different learning abilities on the evolution of the supply chain. The simulation results show that the most appropriate social learning ability can enhance competition between manufacturers and improve product utility. With the enhancement of self-learning ability, manufacturers have a wider range of pricing, products are more diverse, and consumers have a wider choice of goods. This study shows certain guiding significance for the scientific management of the supply chain and the optimization of the competitive strategy of the enterprises on the chain.

Symbiotic Communications: Where Marconi Meets Darwin

With the proliferation of wireless applications, the electromagnetic (EM) space is becoming more and more crowded and complex. This makes it a challenging task to accommodate the growing number of radio systems with limited radio resources. In this article, by considering the EM space as a radio ecosystem, and leveraging the analogy to the natural ecosystem in biology, a novel symbiotic communication (SC) paradigm is proposed through which the relevant radio systems, called symbiotic radios (SRs), in a radio ecosystem form a symbiotic relationship (e.g., mutualistic symbiosis) through intelligent resource/service exchange. Radio resources include, for example, spectrum, energy, and infrastructure, while typical radio services are communicating, relaying, and computing. The symbiotic relationship can be realized via either symbiotic coevolution or symbiotic synthesis. In symbiotic coevolution, each SR is empowered with an evolutionary cycle alongside the multi-agent learning, while in symbiotic synthesis, the SRs ingeniously optimize their operating parameters and transmission protocols by solving a multi-objective optimization problem. Promisingly, the proposed SC paradigm breaks the boundary of radio systems, thus providing us with a fresh perspective on radio resource management and new guidelines to design future wireless communication systems.

QMIX Aided Routing in Social-Based Delay-Tolerant Networks

Delay-tolerant network (DTN) is a network that’s designed to operate effectively in heterogeneous networks that may lack continuous network connectivity. It is characterized by their lack of instantaneous end-to-end paths, resulting in difficulties in designing effective DTN routing protocols. Traditional routing algorithms largely rely on greedy schemes. Such schemes can not guarantee the packets will be eventually transmitted to their destinations, thereby presenting a poor transmission efficiency. Recently, the social-based method has attracted a large amount of attention in wireless network routing. It can use the community and the centrality information to increase the delivery rate of the whole network. Therefore, in this article, we introduce the social-based mechanism to our DTN routing design. Besides, how the distributed nodes can learn the collaboration strategies is another challenge. Inspired by the recent success of multi-agent learning in online control, we adopt a centralized training and distributed execution learning paradigm and design a hierarchical social-based DTN architecture. Based on this, we propose a collaborative multi-agent reinforcement learning (termed as QMIX) aided routing algorithm.

Method for solving social dilemmas based on multi-agent learning and reputation

Many real-world problems contain social dilemmas that express contradictions between individual and public interests. Such tasks can be solved using deep multi-agent learning. This article focuses on reputation-based social dilemma solving and multi-agent reinforcement learning. Target of the work is the development of a method for solving social dilemmas based on deep multi-agent reinforcement learning. A method for solving social dilemmas based on deep multi-agent reinforcement learning is proposed. The difference between the method and the known analogs is based on the use of the agent's reputation as a part of the reward function. Reputation, in turn, depends on the degree of cooperation of an agent with other agents. The software implementation of the method is done in Python using the open library Pytorch. The effectiveness of the method was assessed on the basis of comparison with methods for solving social dilemmas without taking into account cooperation between agents. The results of the work can be used to create software for unmanned vehicles of the city's transport system. A balance between collective and individual interests by solving social dilemmas in a “smart city” can be achieved through the application of the method proposed in the article.

Multi-Agent Learning-Based Nearly Non-Iterative Stochastic Dynamic Transactive Energy Control of Networked Microgrids

Coordination of networked microgrids (MGs) offers a promising solution to utilize distributed resources flexibilities and accommodate renewable energy. This paper studies real-time coordination of distribution system operation (DSO) and MGs considering multivariate uncertainty. Current researches suffer from inadaptability to dynamic system uncertainty, extensive iterations, and dependence on prediction. To fill these gaps, A novel multi-agent learning based stochastic dynamic programming (MASDP) is proposed to obtain the optimal policy for entities. Specifically, transactive energy control (TEC) mechanism, which requires only market-based information interactions, is employed to facilitate coordination between MG and DSO. A data-driven offline self-learning is proposed for entities to learn how to manage resources in response to system uncertainty. After sufficient offline learning, online operation of MASDP can be run in both non-iterative and iterative manners, by which near-optimal/optimal real-time solutions of DSO and MGs can be given sequentially and distributedly. Numerical comparisons with state-of-art policies and TEC algorithms verify the optimality, efficiency, adaptability, and scalability of MASDP.

Practical Network Conditions for the Convergence of Distributed Optimization

The decentralized nature of multi-agent learning often requires continuous information exchange over a (wireless) communication network, in order to accomplish common global objectives. Uncertainty and delay in communication induce large Age of Information (AoI) for data available at the agents, possibly affecting algorithm performance. In order to understand this, one needs communication models that are representative of practical wireless networks. In this paper, we present a representative model based on the Signal-to-Interference-plus-Noise Ratio (SINR) between pairs of agents. Further, we present a novel medium access control (MAC) protocol that is sensitive to local AoI. Our SINR model facilitates the representation of practical dependency effects like shadowing, fading, interference and external noise. The model is driven by an underlying geometrically ergodic Markov chain, which can represent agent mobility. Our MAC protocol enables that the aforementioned dependency effects decay over time. With this dependency decay, we then control the asymptotic growth of the AoI, to facilitate the convergence of distributed algorithms. Finally, we illustrate our ideas by analyzing the distributed stochastic gradient descent scheme that uses delayed communicated data.

Exploration-exploitation in multi-agent learning: Catastrophe theory meets game theory

Exploration-exploitation is a powerful and practical tool in multi-agent learning (MAL); however, its effects are far from understood. To make progress in this direction, we study a smooth analogue of Q-learning. We start by showing that our learning model has strong theoretical justification as an optimal model for studying exploration-exploitation. Specifically, we prove (1) that smooth Q-learning has bounded regret in arbitrary games for a cost model that explicitly balances game-rewards and exploration-costs, i.e., costs from testing potentially suboptimal actions, and (2) that it always converges to the set of quantal-response equilibria (QRE), the standard solution concept for games with bounded rationality, in arbitrary weighted potential games. In our main task, we then turn to measure the effect of exploration on collective system performance. We characterize the geometry of the QRE surface in low-dimensional MAL systems and link our findings with catastrophe (bifurcation) theory. In particular, as the exploration hyperparameter evolves over-time, the system undergoes phase transitions where the number and stability of equilibria can change radically given an infinitesimal change to the exploration parameter. Based on this, we provide a formal theoretical treatment of how tuning the exploration parameter can provably lead to equilibrium selection with both positive as well as negative (and potentially unbounded) effects to system performance.

Dynamical systems as a level of cognitive analysis of multi-agent learning

A dynamical systems perspective on multi-agent learning, based on the link between evolutionary game theory and reinforcement learning, provides an improved, qualitative understanding of the emerging collective learning dynamics. However, confusion exists with respect to how this dynamical systems account of multi-agent learning should be interpreted. In this article, I propose to embed the dynamical systems description of multi-agent learning into different abstraction levels of cognitive analysis. The purpose of this work is to make the connections between these levels explicit in order to gain improved insight into multi-agent learning. I demonstrate the usefulness of this framework with the general and widespread class of temporal-difference reinforcement learning. I find that its deterministic dynamical systems description follows a minimum free-energy principle and unifies a boundedly rational account of game theory with decision-making under uncertainty. I then propose an on-line sample-batch temporal-difference algorithm which is characterized by the combination of applying a memory-batch and separated state-action value estimation. I find that this algorithm serves as a micro-foundation of the deterministic learning equations by showing that its learning trajectories approach the ones of the deterministic learning equations under large batch sizes. Ultimately, this framework of embedding a dynamical systems description into different abstraction levels gives guidance on how to unleash the full potential of the dynamical systems approach to multi-agent learning.

Hierarchical Reinforcement Learning

Hierarchical Reinforcement Learning (HRL) enables autonomous decomposition of challenging long-horizon decision-making tasks into simpler subtasks. During the past years, the landscape of HRL research has grown profoundly, resulting in copious approaches. A comprehensive overview of this vast landscape is necessary to study HRL in an organized manner. We provide a survey of the diverse HRL approaches concerning the challenges of learning hierarchical policies, subtask discovery, transfer learning, and multi-agent learning using HRL. The survey is presented according to a novel taxonomy of the approaches. Based on the survey, a set of important open problems is proposed to motivate the future research in HRL. Furthermore, we outline a few suitable task domains for evaluating the HRL approaches and a few interesting examples of the practical applications of HRL in the Supplementary Material.