Multi-agent Learning Framework Research Articles

An innovative multi-agent approach for robust cyber–physical systems using vertical federated learning

Federated learning presents a compelling approach to training artificial intelligence systems in decentralized settings, prioritizing data safety over traditional centralized training methods. Understanding correlations among higher-level threats exhibiting abnormal behavior in the data stream becomes paramount to developing cyber–physical systems resilient to diverse attacks within a continuous data exchange framework. This work introduces a novel vertical federated multi-agent learning framework to address the challenges of modeling attacker and defender agents in stationary and non-stationary vertical federated learning environments. Our approach uniquely applies synchronous Deep Q-Network (DQN) based agents in stationary environments, facilitating convergence towards optimal strategies. Conversely, in non-stationary contexts, we employ synchronous Advantage Actor–Critic (A2C) based agents, adapting to the dynamic nature of multi-agent vertical federated reinforcement learning (VFRL) environments. This methodology enables us to simulate and analyze the adversarial interplay between attacker and defender agents, ensuring robust policy development. Our exhaustive analysis demonstrates the effectiveness of our approach, showcasing its capability to learn optimal policies in both static and dynamic setups, thus significantly advancing the field of cyber-security in federated learning contexts. To evaluate the effectiveness of our approach, we have done a comparative analysis with its baseline schemes. The findings of our study show significant enhancements compared to the standard methods, confirming the efficacy of our methodology. This progress dramatically enhances the area of cyber-security in the context of federated learning by facilitating the formulation of substantial policies. The proposed scheme attains 15.93%, 32.91%, 31.02%, and 47.26% higher results as compared to the A3C, DDQN, DQN, and Reinforce, respectively.

Deep Reinforcement Multiagent Learning Framework for Information Gathering with Local Gaussian Processes for Water Monitoring

The conservation of hydrological resources involves continuously monitoring their contamination. A multiagent system composed of autonomous surface vehicles is proposed herein to efficiently monitor the water quality. To achieve a safe control of the fleet, the fleet policy should be able to act based on measurements and fleet state. It is proposed to use local Gaussian processes and deep reinforcement learning to jointly obtain effective monitoring policies. Local Gaussian processes, unlike classical global Gaussian processes, can accurately model the information in a dissimilar spatial correlation which captures more accurately the water quality information. A deep convolutional policy is proposed, that bases the decisions on the observation on the mean and variance of this model, by means of an information gain reward. Using a double deep Q‐learning algorithm, agents are trained to minimize the estimation error in a safe manner thanks to a Consensus‐based heuristic. Simulation results indicate an improvement of up to 24% in terms of the mean absolute error with the proposed models. Also, training results with 1–3 agents indicate that our proposed approach returns 20% and 24% smaller average estimation errors for, respectively, monitoring water quality variables and monitoring algae blooms, as compared to state‐of‐the‐art approaches.

Internet Traffic Prediction with Distributed Multi-Agent Learning

Internet traffic prediction has been considered a research topic and the basis for intelligent network management and planning, e.g., elastic network service provision and content delivery optimization. Various methods have been proposed in the literature for Internet traffic prediction, including statistical, machine learning and deep learning methods. However, most of the existing approaches are trained and deployed in a centralized approach, without considering the realistic scenario in which multiple parties are concerned about the prediction process and the prediction model can be trained in a distributed approach. In this study, a distributed multi-agent learning framework is proposed to fill the research gap and predict Internet traffic in a distributed approach, in which each agent trains a base prediction model and the individual models are further aggregated with the cooperative interaction process. In the numerical experiments, two sophisticated deep learning models are chosen as the base prediction model, namely, long short-term memory (LSTM) and gated recurrent unit (GRU). The numerical experiments demonstrate that the GRU model trained with five agents achieves state-of-the-art performance on a real-world Internet traffic dataset collected in a campus backbone network in terms of root mean square error (RMSE) and mean absolute error (MAE).

Multiagent Self-Redundancy Identification and Tuned Greedy-Exploration.

The constant development of sensing applications using innovative and affordable measurement devices has increased the amount of data transmitted through networks, carrying in many cases, redundant information that requires more time to be analyzed or larger storage centers. This redundancy is mainly present because the network nodes do not recognize environmental variations requiring exploration, which causes a repetitive data collection in a set of limited locations. In this work, we propose a multiagent learning framework that uses the Gaussian process regression (GPR) to allow the agents to predict the environmental behavior by means of the neighborhood measurements, and the rate distortion function to establish a border in which the environmental information is neither misunderstood nor redundant. We apply this framework to a mobile sensor network and demonstrate that the nodes can tune the parameter s of the Blahut-Arimoto algorithm in order to adjust the gathered environment information and to become more or less exploratory within a sensing area.

Reward-Reinforced Generative Adversarial Networks for Multi-Agent Systems

Multi-agent systems deliver highly resilient and adaptable solutions for common problems in telecommunications, aerospace, and industrial robotics. However, achieving an optimal global goal remains a persistent obstacle for collaborative multi-agent systems, where learning affects the behaviour of more than one agent. A number of nonlinear function approximation methods have been proposed for solving the Bellman equation, which describe a recursive format of an optimal policy. However, how to leverage the value distribution based on reinforcement learning, and how to improve the efficiency and efficacy of such systems remain a challenge. In this work, we developed a reward-reinforced generative adversarial network to represent the distribution of the value function, replacing the approximation of Bellman updates. We demonstrated our method is resilient and outperforms other conventional reinforcement learning methods. This method is also applied to a practical case study: maximising the number of user connections to autonomous airborne base stations in a mobile communication network. Our method maximises the data likelihood using a cost function under which agents have optimal learned behaviours. This reward-reinforced generative adversarial network can be used as a generic framework for multi-agent learning at the system level.

Distortion based potential game for distributed coverage control

In this work, we propose a multi-agent learning framework to address the mobile sensor coverage problem in which the minimum mutual information between the agents (mobile sensors) and their environment defines the agent-strategy selection rule towards the system Nash equilibrium in a potential game setting. Initially, the agents infer the environment behavior by means of the Gaussian process regression (GPR) approach, using their own information and the information provided by their neighborhood. Then, the rate distortion function (RDF) is used to minimize the mutual information between each agent and its environment by means of the Blahut-Arimoto algorithm, from which, the resulting conditional probability and the parameter λ have the highest relevance, since, the former describes the similitude between the agent information and its environment, and the latter, due to its influence in the distortion and gathered environment information, defines the rationality measure in the learning process. Finally, the expected distortion defined in the RDF, allows the formulation of a distortion based potential function and the consequent equilibrium convergence.

Machine Learning Empowered Spectrum Sharing in Intelligent Unmanned Swarm Communication Systems: Challenges, Requirements and Solutions

The unmanned swarm system (USS) has been seen as a promising technology, and will play an extremely important role in both the military and civilian fields such as military strikes, disaster relief and transportation business. As the “nerve center” of USS, the unmanned swarm communication system (USCS) provides the necessary information transmission medium so as to ensure the system stability and mission implementation. However, challenges caused by multiple tasks, distributed collaboration, high dynamics, ultra-dense and jamming threat make it hard for USCS to manage limited spectrum resources. To tackle with such problems, the machine learning (ML) empowered intelligent spectrum management technique is introduced in this paper. First, based on the challenges of the spectrum resource management in USCS, the requirement of spectrum sharing is analyzed from the perspective of spectrum collaboration and spectrum confrontation. We found that suitable multi-agent collaborative decision making is promising to realize effective spectrum sharing in both two perspectives. Therefore, a multi-agent learning framework is proposed which contains mobile-computing-assisted and distributed structures. Based on the framework, we provide case studies. Finally, future research directions are discussed.

MULTI-AGENT LEARNING FRAMEWORK FOR ENVIRONMENT REDUNDANCY IDENTIFICATION FOR MOBILE SENSORS IN AN IOT CONTEXT

Abstract. From an IoT point of view, the continuous growth of cheap and versatile sensor technologies has generated a massive data flow in communication networks, which most of the time carries unnecessary or redundant information that requires larger storage centers and more time to process and analyze data. Most of this redundancy is due to fact that network nodes are unable to identify environmental cues showing measurement changes to be considered and instead remain at a static location getting the same data. In this work we propose a multi-agent learning framework based on two theoretical tools. Firstly, we use Gaussian Process Regression (GPR) to make each node capable of getting information from the environment based on its current measurement and the measurements taken by its neighbors. Secondly, we use the rate distortion function to define a boundary where the information coming from the environment is neither redundant nor misunderstood. Finally, we show how the framework is applied in a mobile sensor network in which sensors decide to be more or less exploratory by means of the parameter s of the Blahut-Arimoto algorithm, and how it affects the measurement coverage in a spatial area being sensed.

COOPERATIVE MULTI-AGENT LEARNING IN SOCIAL NETWORKS: A REVIEW

Coordination in cooperative multi-agent systems is one of the important issues in multi-agent learning and has been broadly studies in the literature. Much work in this field has been carried out yet. However, there are still some coordination issues that are required to be improved. In this work, we look over the multi-agent coordination problems in cooperative environments under the networked multi-agent learning framework using some social network structures and will try to improve coordination efficiency. In our framework, we talk about two types of learners, i.e. individual action learner and joint action learner. We are considering hierarchical multi-agent learning framework to accelerate the coordination efficiency. Our research direction is to consider utilizing the characteristics of different neighboring agents (e.g., the past coordination histories of different nodes, the relative degree of different nodes in the neighborhood) while performing the multi-agent learning to improve coordination performance.

MALEF: Framework for distributed machine learning and data mining

Growing importance of distributed data mining techniques has recently attracted attention of researchers in multiagent domain. In this paper we present a novel framework MultiAgent Learning Framework (MALEF) designed for both the agent-based distributed machine learning as well as data mining. Proposed framework is based on: the exchange of meta-level descriptions of individual learning process; online reasoning about learning success and learning progress. This paper illustrates how MALEF framework can be used in practical system in which different learners use different datasets, hypotheses and learning algorithms. We describe our experimental results obtained using this system and review related work on the subject.

Multi-agent learning for engineers

As suggested by the title of Shoham, Powers, and Grenager's position paper [Y. Shoham, R. Powers, T. Grenager, If multi-agent learning is the answer, what is the question? Artificial Intelligence 171 (7) (2007) 365–377, this issue], the ultimate lens through which the multi-agent learning framework should be assessed is “what is the question?”. In this paper, we address this question by presenting challenges motivated by engineering applications and discussing the potential appeal of multi-agent learning to meet these challenges. Moreover, we highlight various differences in the underlying assumptions and issues of concern that generally distinguish engineering applications from models that are typically considered in the economic game theory literature.