Multi-agent Learning Research Articles

Interactive multi-agent convolutional broad learning system for EEG emotion recognition

Electroencephalogram (EEG) emotion recognition is gaining significance in intelligent human–computer interaction. Multi-agent learning can capture more complete and reliable features, compensating for the lack of a single agent’s knowledge. However, previous EEG emotion recognition only focused on a single agent. Therefore, to extract effective features from high-dimensional EEG data, a novel interactive multi-agent (IMA) learning framework is proposed, and introduced into convolutional broad learning system (CNNBLS), then the interactive multi-agent convolutional broad learning system (IMA-CNNBLS) is proposed for EEG emotion recognition. It can effectively model high-dimensional EEG data, automatically extract EEG features related to emotions through convolutional neural network (CNN), and then extend the above features to a vast space to quickly extract generalized features through broad learning system (BLS), consider a CNNBLS as an agent, and multi-agent interact in the IMA part. As the theoretical basis of IMA-CNNBLS, the consistency is mathematically proved through the stationary distribution theory of Markov processes. To demonstrate the superiority of our proposed model, sufficient experiments are conducted on the DEAP, DREAMER and SEED datasets. The experimental results show the model can effectively improve the EEG emotion recognition accuracy, the best recognition performance is achieved on all datasets. In addition, our proposed model also shows the multi-agent interaction significantly affects EEG emotion recognition.

Reliable and Energy-Efficient Communications in Mobile Robotic Networks by Collaborative Beamforming

For mobile robotic networks in industrial scenarios, reliable and energy-efficient communications are crucial yet challenging. Fortunately, collaborative beamforming (CB) emerges as a promising solution, which can increase the transmission gain and reduce the transmit power of robots by constructing a mobile robot-enabled virtual antenna array (MRVAA). The performance of CB is tightly related to robot positions, necessitating proper robot selection. However, robot selection may expose the network to the risk of unbalanced energy distribution, reducing network lifetime. Additionally, the mobility and variable numbers of robots require flexible and scalable robot selection algorithms. To tackle these challenges, we first formulate a multi-objective optimization problem to reduce the maximum sidelobe level (MSLL) of MRVAA while minimizing the standard deviation of the network energy distribution (SDNED) by selecting robots for CB. Then, based on distributed multi-agent learning (MARL), we propose an effective and scalable robot selection algorithm with energy considered (RoSE) to solve the problem, where difference-rewards function (DRF) and policy sharing are designed for enhancing convergence rate and policy stability. Simulation results show that the RoSE has the scalability to positions and numbers of robots. Furthermore, RoSE surpasses existing selection algorithms in network lifetime and time efficiency, while still maintaining comparable MSLL.

Deep Reinforcement Learning Versus Evolution Strategies: A Comparative Survey.

Deep reinforcement learning (DRL) and evolution strategies (ESs) have surpassed human-level control in many sequential decision-making problems, yet many open challenges still exist. To get insights into the strengths and weaknesses of DRL versus ESs, an analysis of their respective capabilities and limitations is provided. After presenting their fundamental concepts and algorithms, a comparison is provided on key aspects, such as scalability, exploration, adaptation to dynamic environments, and multiagent learning. Current research challenges are also discussed, including sample efficiency, exploration versus exploitation, dealing with sparse rewards, and learning to plan. Then, the benefits of hybrid algorithms that combine DRL and ESs are highlighted.

Safe Multiagent Learning With Soft Constrained Policy Optimization in Real Robot Control

Safe Multiagent Learning With Soft Constrained Policy Optimization in Real Robot Control

Multiexperience-Assisted Efficient Multiagent Reinforcement Learning.

Recently, multiagent reinforcement learning (MARL) has shown great potential for learning cooperative policies in multiagent systems (MASs). However, a noticeable drawback of current MARL is the low sample efficiency, which causes a huge amount of interactions with environment. Such amount of interactions greatly hinders the real-world application of MARL. Fortunately, effectively incorporating experience knowledge can assist MARL to quickly find effective solutions, which can significantly alleviate the drawback. In this article, a novel multiexperience-assisted reinforcement learning (MEARL) method is proposed to improve the learning efficiency of MASs. Specifically, monotonicity-constrained reward shaping is innovatively designed using expert experience to provide additional individual rewards to guide multiagent learning efficiently, with the invariance guarantee of the team optimization objective. Furthermore, a reward distribution estimator is specially developed to model an implicated reward distribution of environment by using transition experience from environment, containing collected samples (state-action pair, reward, and next state). This estimator can predict the expectation reward of each agent for the taken action to accurately estimate the state value function and accelerate its convergence. Besides, the performance of MEARL is evaluated on two multiagent environment platforms: our designed unmanned aerial vehicle combat (UAV-C) and StarCraft II Micromanagement (SCII-M). Simulation results demonstrate that the proposed MEARL can greatly improve the learning efficiency and performance of MASs and is superior to the state-of-the-art methods in multiagent tasks.

Multiagent learning for competitive opinion optimization

From a perspective of designing or engineering for opinion formation games in social networks, the opinion maximization (or minimization) problem has been studied mainly for designing seeding algorithms that aim at selecting a subset of nodes to control their opinions. We first define a two-player zero-sum Stackelberg game of competitive opinion optimization by letting the player under study as the leader minimize the sum of expressed opinions by doing so-called “internal opinion design”, knowing that the other adversarial player as the follower is to maximize the same objective by also conducting her own internal opinion design. We furthermore consider multiagent learning, specifically using the Optimistic Gradient Descent Ascent, and analyze its convergence to equilibria in the simultaneous-game version of competitive opinion optimization.

Authentication communication by using visualization cryptography for UAV networks

Utilizing unmanned aerial vehicles (UAVs) to support V2X requirements leverages their versatility and line-of-sight communication. Our prior work explored a multi-agent learning approach for resource optimization, maximizing task offloading while maintaining QoS. This paper focuses on securing UAV communication, particularly authentication. Traditional methods are often unsuitable due to UAV limitations. We propose a novel authentication mechanism for a single ground control station (GCS) interacting with multiple UAVs across flight sessions. The system utilizes a key generation method based on chaotic maps to create unique flight session keys for each pre-defined flight plan. These keys, along with flight plans, are registered in a secure database. During flight, the GCS verifies UAV identity by employing the flight session key and corresponding flight plan for message authentication. This approach reduces computational and communication overhead compared to traditional certificate exchanges and asymmetric cryptography, which are energy-intensive for UAVs. While not a comprehensive security solution, this method provides an initial layer of protection for the UAV network.

GLIDE: Multi-Agent Deep Reinforcement Learning for Coordinated UAV Control in Dynamic Military Environments

Unmanned aerial vehicles (UAVs) are widely used for missions in dynamic environments. Deep Reinforcement Learning (DRL) can find effective strategies for multiple agents that need to cooperate to complete the task. In this article, the challenge of controlling the movement of a fleet of UAVs is addressed by Multi-Agent Deep Reinforcement Learning (MARL). The collaborative movement of the UAV fleet can be controlled centrally and also in a decentralized fashion, which is studied in this work. We consider a dynamic military environment with a fleet of UAVs, whose task is to destroy enemy targets while avoiding obstacles like mines. The UAVs inherently come with a limited battery capacity directing our research to focus on the minimum task completion time. We propose a continuous-time-based Proximal Policy Optimization (PPO) algorithm for multi-aGent Learning In Dynamic Environments (GLIDE). In GLIDE, the UAVs coordinate among themselves and communicate with the central base to choose the best possible action. The action control in GLIDE can be controlled in a centralized and decentralized way, and two algorithms called Centralized-GLIDE (C-GLIDE), and Decentralized-GLIDE (D-GLIDE) are proposed on this basis. We developed a simulator called UAV SIM, in which the mines are placed at randomly generated 2D locations unknown to the UAVs at the beginning of each episode. The performance of both the proposed schemes is evaluated through extensive simulations. Both C-GLIDE and D-GLIDE converge and have comparable performance in target destruction rate for the same number of targets and mines. We observe that D-GLIDE is up to 68% faster in task completion time compared to C-GLIDE and could keep more UAVs alive at the end of the task.

Transformer-Based Reinforcement Learning for Multi-Robot Autonomous Exploration.

A map of the environment is the basis for the robot's navigation. Multi-robot collaborative autonomous exploration allows for rapidly constructing maps of unknown environments, essential for application areas such as search and rescue missions. Traditional autonomous exploration methods are inefficient due to the repetitive exploration problem. For this reason, we propose a multi-robot autonomous exploration method based on the Transformer model. Our multi-agent deep reinforcement learning method includes a multi-agent learning method to effectively improve exploration efficiency. We conducted experiments comparing our proposed method with existing methods in a simulation environment, and the experimental results showed that our proposed method had a good performance and a specific generalization ability.

A multi-agent learning framework for mixed-integer linear programming

Mixed integer linear programming (MILP) is an important problem in the combinatorial optimization domain, which has wide applications in practical optimization scenarios. Given that most MILP problems fall into the NP-hard category, which the traditional methods may fail to solve, recent research has tried to derive MILP solutions using machine learning techniques. The whole MILP-solving procedure involves lots of modules, such as pre-solving, cut selection, node section, etc., and these modules are closely related and influence each other. However, the previous machine learning-based approaches neglect the connections between these modules, and focus on single-module learning techniques. To address this, we propose an initial step towards a more comprehensive multi-agent learning framework that allows different modules to interact and collaborate. Specifically, our current implementation involves two key modules: HEM for cut selection applied at the root node and GCNN for variable selection. By employing HEM to influence the training of GCNN, these two agents thus work in unison. Through extensive experiments on four MILP datasets in diverse scenarios, we observe significant improvements in solving time and PD integral metrics compared with the state-of-the-art learning-based MILP solving methods. This work lays the groundwork for future development of a fully integrated multi-agent framework.

An innovative multi-agent approach for robust cyber–physical systems using vertical federated learning

Federated learning presents a compelling approach to training artificial intelligence systems in decentralized settings, prioritizing data safety over traditional centralized training methods. Understanding correlations among higher-level threats exhibiting abnormal behavior in the data stream becomes paramount to developing cyber–physical systems resilient to diverse attacks within a continuous data exchange framework. This work introduces a novel vertical federated multi-agent learning framework to address the challenges of modeling attacker and defender agents in stationary and non-stationary vertical federated learning environments. Our approach uniquely applies synchronous Deep Q-Network (DQN) based agents in stationary environments, facilitating convergence towards optimal strategies. Conversely, in non-stationary contexts, we employ synchronous Advantage Actor–Critic (A2C) based agents, adapting to the dynamic nature of multi-agent vertical federated reinforcement learning (VFRL) environments. This methodology enables us to simulate and analyze the adversarial interplay between attacker and defender agents, ensuring robust policy development. Our exhaustive analysis demonstrates the effectiveness of our approach, showcasing its capability to learn optimal policies in both static and dynamic setups, thus significantly advancing the field of cyber-security in federated learning contexts. To evaluate the effectiveness of our approach, we have done a comparative analysis with its baseline schemes. The findings of our study show significant enhancements compared to the standard methods, confirming the efficacy of our methodology. This progress dramatically enhances the area of cyber-security in the context of federated learning by facilitating the formulation of substantial policies. The proposed scheme attains 15.93%, 32.91%, 31.02%, and 47.26% higher results as compared to the A3C, DDQN, DQN, and Reinforce, respectively.

Cournot Policy Model: Rethinking centralized training in multi-agent reinforcement learning

This work studies Centralized Training and Decentralized Execution (CTDE), which is a powerful mechanism to ease multi-agent reinforcement learning. Although the centralized evaluation ensures unbiased estimates of Q-value, peers with unknown policies make the decentralized policy far from the expectation. To make progress in more stabilized and effective joint policy, we develop a novel game framework, termed Cournot Policy Model, to enhance the CTDE-based multi-agent learning. Combining the game theory and reinforcement learning, we regard the joint decision-making in a single time step as a Cournot duopoly model, and then design a Hetero Variational Auto-Encoder to model the policies of peers in the decentralized execution. With a conditional policy, each agent is guided to a stable mixed-strategy equilibrium even though the joint policy evolves over time. We further demonstrate that such an equilibrium must exist in the case of centralized evaluation. We investigate the improvement of our method on existing centralized learning methods. The experimental results on a comprehensive collection of benchmarks indicate our approach consistently outperforms baseline methods.

Interpersonal trust: Asymptotic analysis of a stochastic coordination game with multi-agent learning

Interpersonal trust: Asymptotic analysis of a stochastic coordination game with multi-agent learning

Cooperative Multi-Agent Learning for Navigation via Structured State Abstraction

Cooperative Multi-Agent Learning for Navigation via Structured State Abstraction

Mathematics of multi-agent learning systems at the interface of game theory and artificial intelligence

Mathematics of multi-agent learning systems at the interface of game theory and artificial intelligence

Deep Reinforcement Multiagent Learning Framework for Information Gathering with Local Gaussian Processes for Water Monitoring

The conservation of hydrological resources involves continuously monitoring their contamination. A multiagent system composed of autonomous surface vehicles is proposed herein to efficiently monitor the water quality. To achieve a safe control of the fleet, the fleet policy should be able to act based on measurements and fleet state. It is proposed to use local Gaussian processes and deep reinforcement learning to jointly obtain effective monitoring policies. Local Gaussian processes, unlike classical global Gaussian processes, can accurately model the information in a dissimilar spatial correlation which captures more accurately the water quality information. A deep convolutional policy is proposed, that bases the decisions on the observation on the mean and variance of this model, by means of an information gain reward. Using a double deep Q‐learning algorithm, agents are trained to minimize the estimation error in a safe manner thanks to a Consensus‐based heuristic. Simulation results indicate an improvement of up to 24% in terms of the mean absolute error with the proposed models. Also, training results with 1–3 agents indicate that our proposed approach returns 20% and 24% smaller average estimation errors for, respectively, monitoring water quality variables and monitoring algae blooms, as compared to state‐of‐the‐art approaches.

A GNN-based teacher–student framework with multi-advice

Multi-agent learning involves the interaction of multiple agents with the environment to learn an optimal policy. To enhance learning performance, a commonly used approach is the teacher–student framework, which enables agents to seek advice from other agents. However, the current literature suffers from a common limitation, wherein a student agent can only receive advice from a single teacher agent at each time step, thus constraining the knowledge acquired. Although some methods allow advice from multiple teachers, they require pre-training of the teachers, which is impractical when agents concurrently learn from scratch simultaneously. Additionally, the importance of the agents’ connection structures is often disregarded, despite its critical role in the advice-giving process. To overcome these limitations, we propose a novel advising approach utilizing graph neural networks (GNNs). This method models the agents’ connection structures, learns the weight of advice, and aggregates inputs from multiple teachers to generate refined advice. Our experiments show that this proposed approach has superior learning performance compared to advising baseline methods.

Blockchain and Multi-Agent Learning Empowered Incentive IRS Resource Scheduling for Intelligent Reconfigurable Networks

Blockchain and Multi-Agent Learning Empowered Incentive IRS Resource Scheduling for Intelligent Reconfigurable Networks

A Multiagent Cooperative Learning System With Evolution of Social Roles

Recent developments in reinforcement learning have been able to derive optimal policies for sophisticated and capable agents, and shown to achieve human-level performance on a number of challenging tasks. Unfortunately, when it comes to multi-agent systems (MASs), complexities such as non-stationarity and partial observability bring new challenges to the field. Building a flexible and efficient multi-agent reinforcement learning algorithm capable of handling complex tasks has to date remained an open challenge. This paper presents an Multi-Agent learning system with the evolution of Social Roles (eSRMA). The main interest is placed on solving the key issues in the definition and evolution of suitable roles, and optimizing the policies accompanied by social roles in MAS efficiently. Specifically, eSRMA incorporates and cultivates role division awareness of agents to improve the ability to deal with complex cooperative tasks. Each agent is assigned a role module, which can dynamically generate roles based on the individuals’ local observations. A novel multi-agent reinforcement learning algorithm (MARL) is designed as the principal driving force that governs the role-policy learning process by a role-attention credit assignment mechanism. Moreover, a role evolution process is developed to help agents dynamically choose appropriate roles in decision-making. Comprehensive experiments on the StarCraft II micromanagement benchmark have demonstrated that eSRMA exhibits superiority in achieving higher learning capability and efficiency for multiple agents compared to the state-of-the-art MARL methods.

To Interfere or Not To Interfere: Information Revelation and Price-Setting Incentives in a Multiagent Learning Environment

Demand uncertainty and seller competition are substantial challenges for online platforms. In “To Interfere or Not To Interfere: Information Revelation and Price-Setting Incentives in a Multiagent Learning Environment,” Birge, Chen, Keskin, and Ward analyze whether and how an online platform should offer demand information or price incentives to the sellers participating on the platform. The authors show that, when facing uncertain demand, the platform could be better off by doing nothing—that is, not providing any information or incentives to the sellers. They also develop a strategic reveal-and-incentivize policy for the platform to choose when to start sharing information and offering rewards to coordinate the sellers’ pricing. They prove that the strategic reveal-and-incentivize policy achieves near-optimal profit performance for the platform.