Multi-agent System Model Research Articles

Discovering architecture-aware and sound process models of multi-agent systems: a compositional approach

Discovering architecture-aware and sound process models of multi-agent systems: a compositional approach

Verification of agent navigation in partially-known environments

This paper establishes a framework based on logic and automata theory in which to model and automatically verify systems of multiple mobile agents moving in environments with partially-known topologies, i.e., ones which are not completely known at design time. Examples include physical agents designed to be used in many spatial environments and not tailored for a specific one, robots in environments not reachable by humans, and software exploring partially-mapped networks. We model spatial environments as graphs whose edges are labelled with directions. We model agents as finite-state machines that move on the graphs by issuing commands of the form “go in direction X”, that can communicate their internal state to other agents, and that can sense agent positions (including current and visited positions). We treat the incomplete information about the spatial environment by studying the decision problem that asks whether a given collection of agents achieve their tasks on all graphs from a class of graphs — this is called the parameterised verification problem. The framework also introduces a new logical language based on Linear Temporal Logic that is tailored for expressing agent navigation tasks in such environments.Although the parameterised verification problem is undecidable, we identify two key dimensions that need to be limited in order to regain decidability, namely, the set of graph-environments and the amount of sensing and communication between agents. In particular, one should limit the families of graphs to exclude grids, and there should be a bound on the number of times an agent senses the position of another agent or communicates its own state to another agent. We prove that dropping either of these assumptions results in undecidability, even for agents with severe restrictions on their abilities (e.g., with very limited sensing abilities and no communication abilities). The importance of this work is that a) it provides a general computational model for mobile multi-agent systems in environments with partially-known topologies, b) it identifies, for the first time, the precise causes of undecidability of these systems and presents minimal restrictions to alleviate this problem, and c) it provides a generic sound and complete procedure for solving the parameterised verification problem over a broad range of spatial-environments and for agents with very powerful sensing and communication abilities.

Multi-Agent Deep Q Network to Enhance the Reinforcement Learning for Delayed Reward System

This study examines various factors and conditions that are related with the performance of reinforcement learning, and defines a multi-agent DQN system (N-DQN) model to improve them. N-DQN model is implemented in this paper with examples of maze finding and ping-pong as examples of delayed reward system, where delayed reward occurs, which makes general DQN learning difficult to apply. The implemented N-DQN shows about 3.5 times higher learning performance compared to the Q-Learning algorithm in the reward-sparse environment in the performance evaluation, and compared to DQN, it shows about 1.1 times faster goal achievement speed. In addition, through the implementation of the prioritized experience replay and the implementation of the reward acquisition section segmentation policy, such a problem as positive-bias of the existing reinforcement learning models seldom or never occurred. However, according to the characteristics of the architecture that uses many numbers of actors in parallel, the need for additional research on light-weighting the system for further performance improvement has raised. This paper describes in detail the structure of the proposed multi-agent N_DQN architecture, the contents of various algorithms used, and the specification for its implementation.

Simulating spatiotemporal dynamics of urban underground space development using multi-agent system: A case study in Changzhou City, China

The development of urban underground space (UUS) serves to promote the intensive use of land resources, overcome the shortage of space resources, and achieve sustainable urban development. Since UUS development is a dynamic and complex process involving diverse stakeholders, investigating its operating mechanisms and predicting its trajectory is crucial to achieve sustainable urban development. This study used a multi-agent system modeling method to build a UUS development decision-making model for exploring the interactions between different stakeholders and their adaptive decision-making behaviors. The model allows the dynamic UUS development process to be presented spatially, considering the spatiotemporal changes in environmental, economic, and social factors. The actual status and three future scenarios of UUS development were simulated and analyzed for the Yanling business district of Changzhou City, China. The simulation results show that the model can effectively describe, interpret, and predict the dynamic spatiotemporal process of UUS development, thereby providing theoretical reference and decision-making support for UUS planning and policymaking.

H ∞ Consensus Design and Online Scheduling for Multiagent Systems with Switching Topologies via Deep Reinforcement Learning

This paper is devoted to H ∞ consensus design and online scheduling for homogeneous multiagent systems (MASs) with switching topologies via deep reinforcement learning. The model of homogeneous MASs with switching topologies is established based on switched systems theory, in which the switching of topologies is viewed as the switching among subsystems. By employing linear transformation, the closed-loop systems of MASs are converted into reduced-order systems. The problem of H ∞ consensus design can be transformed to the issue of H ∞ control. It is supposed that the consensus protocol is composed of two parts: dynamics-based protocol and learning-based protocol, where dynamics-based protocol is provided to guarantee the convergence and weighted attenuation and learning-based protocol is proposed to improve the transient performance. Then, the multiple Lyapunov function (MLF) method and mode-dependent average dwell time (MDADT) method are combined to ensure the stability and weighted H ∞ disturbance attenuation index of reduced-order systems. The sufficient existing conditions of dynamics-based protocol are given through the feasible solutions of linear matrix inequalities (LMIs). Moreover, the online scheduling is formulated as a Markov decision process, and the deep deterministic policy gradient (DDPG) algorithm in the framework of actor-critic is proposed for the compensation of disturbance to explore optimal control policy. The online scheduling of parameters of MASs is viewed as bounded compensation of dynamics-based protocol, whose stability can be guaranteed by nonfragile control theory. Finally, simulation results are provided to illustrate the effectiveness and superiority of the proposed method.

ИНТЕЛЛЕКТУАЛИЗАЦИЯ УПРАВЛЕНИЯ МУЛЬТИАГЕНТНЫМ ВЗАИМОДЕЙСТВИЕМ ПРИ ОРГАНИЗАЦИИ ГРУЗОВЫХ ПЕРЕВОЗОК В ПРИПОРТОВЫХ ТРАНСПОРТНЫХ СИСТЕМАХ

Under the conditions of operation's and rolling stock owner’s plurality on the Russian railwaynetwork the following issues are in case: excessive freight-hauling and carrying capacity ofsection, oncoming rerun of empty boxcar in one type, excessive mileage of empty boxcars, sectionalspeed reduction and others. More effective cooperation of transit process members based onlogistics, market simulation of freight traffics, creation of math models with the use of digitalizationand intellectualization methods of control are necessary for solving mentioned issues. Thisstudy is devoted to research matters of principles updating modelling multi-agent interaction inthe portside transport systems. Methods of statistical, morphological, regression and system analysis,mathematical and analytical modelling compose the methodological basis of the research.Research and modelling characteristics of freight and car traffic flow distribution under contextsof multiagency of transport complex, wherein the author-developed economic and geographicalmethod to delimit stations’ «spheres of influence» is applied allow to create analytical models oftransportation process based on integrated assessment of transport and technological infrastructureof the railway polygon and cost of transport services. Building of a digital geographical modelof rolling stock distribution according to types of transportation service for portside stations byanalytic curves of higher order is one of special characteristics. «Spheres of influence» of loadingstations obtained by methods of economic and geographical delimitation allows formulating arange of preferential directions while distributing car traffic. Acquired territorial scene of distributingcar traffic is a basis for solution of multi-attribute problem of optimization of boxcar directionsregulation taking into account multi-operator market of rolling stock, digitalization and intellectualizationof the branch. In addition to solution of objectives of transport services marketregulation in terms of rolling stock distribution, the problems of technological, economic, fiscaland digital interaction on a basis of logistics in the multi-agent systems are continued to be crucial.Model and methodological propositions formed in this context should ensure reduction oftransport and logistic costs with parallel improvement of quantitative, qualitative and temporaryindicators of integrated logistic supply chains realization.

Intelligent L2-L∞ Consensus of Multiagent Systems under Switching Topologies via Fuzzy Deep Q Learning.

The problem of intelligent L2-L∞ consensus design for leader-followers multiagent systems (MASs) under switching topologies is investigated based on switched control theory and fuzzy deep Q learning. It is supposed that the communication topologies are time-varying, and the model of MASs under switching topologies is constructed based on switched systems. By employing linear transformation, the problem of consensus of MASs is converted into the issue of L2-L∞ control. The consensus protocol is composed of the dynamics-based protocol and learning-based protocol, where the robust control theory and deep Q learning are applied for the two parts to guarantee the prescribed performance and improve the transient performance. The multiple Lyapunov function (MLF) method and mode-dependent average dwell time (MDADT) method are combined to give the scheduling interval, which ensures stability and prescribed attenuation performance. The sufficient existing conditions of consensus protocol are given, and the solutions of the dynamics-based protocol are derived based on linear matrix inequalities (LMIs). Then, the online design of the learning-based protocol is formulated as a Markov decision process, where the fuzzy deep Q learning is utilized to compensate for the uncertainties and achieve optimal performance. The variation of the learning-based protocol is modeled as the external compensation on the dynamics-based protocol. Therefore, the convergence of the proposed protocol can be guaranteed by employing the nonfragile control theory. In the end, a numerical example is given to validate the effectiveness and superiority of the proposed method.

Bayesian Optimization of Distributed Neurodynamical Controller Models for Spatial Navigation

Dynamical systems models for controlling multi-agent swarms have demonstrated advances toward resilient, decentralized navigation algorithms. We previously introduced the NeuroSwarms controller, in which agent-based interactions were modeled by analogy to neuronal network interactions, including attractor dynamics and phase synchrony, that have been theorized to operate within hippocampal place-cell circuits in navigating rodents. This complexity precludes linear analyses of stability, controllability, and performance typically used to study conventional swarm models. Further, tuning dynamical controllers by hand or grid search is often inadequate due to the complexity of objectives, dimensionality of model parameters, and computational costs of simulation-based sampling. Here, we present a framework for tuning dynamical controller models of autonomous multi-agent systems based on Bayesian Optimization (BayesOpt). Our approach utilizes a task-dependent objective function to train Gaussian Processes (GPs) as surrogate models to achieve adaptive and efficient exploration of a dynamical controller model's parameter space. We demonstrate this approach by studying an objective function selecting for NeuroSwarms behaviors that cooperatively localize and capture spatially distributed rewards under time pressure. We generalized task performance across environments by combining scores for simulations in distinct geometries. To validate search performance, we compared high-dimensional clustering for high- vs. low-likelihood parameter points by visualizing sample trajectories in Uniform Manifold Approximation and Projection (UMAP) embeddings. Our findings show that adaptive, sample-efficient evaluation of the self-organizing behavioral capacities of complex systems, including dynamical swarm controllers, can accelerate the translation of neuroscientific theory to applied domains.

Reliable Task Planning of Networked Devices as a Multi-Objective Problem Using NSGA-II and Reinforcement Learning

Tracking and detecting multiple objects is difficult for a single radar device, as it may not have the capacities such as anti-interference and anti-stealth. However, if radar devices of diverse capabilities can be combined to realize collaborative networked operation, the reliability and performance of a radar system in a complex environment can be significantly improved. This paper classify the networked radar-based multi-objective task planning as a combinatorial optimization problem with constraints and abstract a distributed multi-agent system (MAS) model from a networked radar system. A node-selection algorithm was designed based on a greedy policy to narrow down solution space for subsequent networked radar task planning, reduce the amount of calculation, and improve the efficiency of the proposed algorithm. Moreover, focusing on NSGA-II, the proposed algorithm was modified using self-adaptive operators and reinforcement learning. A dual-population strategy was introduced to allow exchanges of multiple individuals between populations during migration, and the number of individuals for the exchange was obtained through reinforcement learning. In this paper, five algorithms are compared and analyzed. In addition, statistical analyses are conducted from four perspectives: the average evaluation value of energy consumption and bandwidth in the Pareto front solutions, the time consumption of the algorithm, and the diversity of the population. The results indicate that among those algorithms, the reinforcement-learning-based RNSGA-II algorithm can produce the best outcomes for networked radar task planning.

A state-based multi-agent system model of taxi fleets

A state-based multi-agent system model of taxi fleets

Multiagent System and Rainfall-Runoff Model in Hydrological Problems: A Systematic Literature Review

Typically, hydrological problems require approaches capable of describing and simulating part of the hydrological system, or the environmental consequences of natural or anthropic actions. Tools such as Multiagent System (MAS) and Rainfall-Runoff Model (RRM) have been used to help researchers to develop and better understand water systems. Thus, this study presents a Systematic Literature Review (SLR) on the joint use of MAS and RRM tools, in the context of hydrological problems. SLR was performed based on a protocol defined from the research question. Initially, 79 papers were found among six bibliographic databases. This total was reduced over four stages of selection, according to exclusion criteria. In the end, three papers were considered satisfactory within the scope of the research, where they were summarized, analyzed, and compared. While the MAS and RRM tools can interact with their results in a coupled model, SLR showed that there are still major challenges to be explored concerning the dynamics between them, as the steps of scales and validation. However, the coupling of MAS and RRM can provide an interesting alternative tool to analyse decision-making about water resources management systems.

Simulation of online food ordering delivery strategies using multi-agent system models

ABSTRACT With the rapid development of Online to Offline (O2O) business, millions of transactions each day along with the varying processing time of merchants and the complexity of traffic conditions pose significant challenges to effective and efficient delivery of orders. This paper studies the complex adaptive dynamics of O2O platforms by combining the behaviors of customers, merchants, dispatcher, and couriers in the context of a multi-agent model. Serving as a testbed, the simulation model enables the evaluation of alternative order delivery strategies. Preliminary experimental results show that TSP-based delivery strategy is more efficient than the nearer merchant assignment strategy. As an important property, the larger load capacity of couriers is beneficial to improve the completion rate of orders rather than the completion time. Finally, the experiment using the real road network and the real order data demonstrates the applicability of the proposed multi-agent model of O2O platform in the real scenario.

Uncertainty modeling of household appliance loads for smart energy management

Households represent a large share of flexibility on the demand side for assisting the smooth operation of the highly renewable grid. The stochastic nature of activities and various appliance usage patterns of household occupants affects the flexibility potential greatly, but is usually oversimplified in the existing research. This paper aims to propose a novel method for simulating the dynamics of household energy-related activities and appliance usage, which provides realistic synthetic load profiles for smart energy management. To this aim, a highly resolved multi-agent system model is proposed, which comprises an Agent-Based Activity Chain Model (ABACM) and 34 types of common household appliance models. Firstly, the patterns of various occupant behaviors are obtained through mining of real-world residential time-of-use data. Then, the stochastic activity profiles of occupants are generated using the ABACM, with a root mean square error of 0.95%. Finally, the electricity consumption profiles of household appliances are simulated based on specific energy-related activities. The proposed method is validated and proved to be able to capture stochastic occupant behaviors and represent the dynamics of residential energy consumption. The future work will focus on using the proposed model for exploring the potential of residential demand response.

Task Allocation on Layered Multiagent Systems: When Evolutionary Many-Objective Optimization Meets Deep Q-Learning

This article is concerned with the multitask multiagent allocation problem via many-objective optimization for multiagent systems (MASs). First, a novel layered MAS model is constructed to address the multitask multiagent allocation problem that includes both the original task simplification and the many-objective allocation. In the first layer of the model, the deep Q-learning method is introduced to simplify the prioritization of the original task set. In the second layer of the model, the modified shift-based density estimation (MSDE) method is put forward to improve the conventional strength Pareto evolutionary algorithm 2 (SPEA2) in order to achieve many-objective optimization on task assignments. Then, an MSDE-SPEA2-based method is proposed to tackle the many-objective optimization problem with objectives including task allocation, makespan, agent satisfaction, resource utilization, task completion, and task waiting time. As compared with the existing allocation methods, the developed method in this article exhibits an outstanding feature that the task assignment and the task scheduling are carried out simultaneously. Finally, extensive experiments are conducted to: 1) verify the validity of the proposed model and the effectiveness of two main algorithms and 2) illustrate the optimal solution for task allocation and efficient strategy for task scheduling under different scenarios.

Integrated Water-Power System Resilience Analysis in a Southeastern Idaho Irrigation District: Minidoka Case Study

This study investigates the joint water–power system resilience of an irrigation district in southeastern Idaho. Irrigation districts face difficulties in the delivery of water to farmers under drought conditions, during equipment failures, or unplanned infrastructure disruptions. The resilience of interconnected water and power systems can be better analyzed and understood through an integrated approach, using a model that connects the dependencies between the two halves of the system. Using a multi-agent system model capturing both water and power system components, as well as their linkages, we capture the interdependencies of these systems and highlight opportunities for improvement when faced with disruptions. Through simulation scenarios, we examine the system resilience using system performance, quantified as the percentage of met demand of the power and water system, when subjected to drought water year, an unforeseen water demand increase, power outage and dam failure. Scenario results indicate that the effects of low flow years are mostly felt in the power system; unexpected increases in water demand marginally impact irrigation system performance; dams and pumps present vulnerabilities of the system, causing substantial unmet demand during disruptions. Noting the interdependencies between the water–power system halves while leveraging an integrated simulation allows for an insightful analysis of the system impacts during disruptions.

Optimal consensus control for unknown second-order multi-agent systems: Using model-free reinforcement learning method

In this paper, the optimal consensus control problem with second-order dynamics consisting of leader and follower agents is discussed. For optimal consensus problem, the optimal control policies rely on algebraic Riccati equations (AREs) equation, which are difficult to solve. Furthermore, both the follower agents’ and the leader agent’s dynamics are assumed to be completely unknown. As the consensus problem based on feedback control, the second-order discrete-time multi-agent systems (DT-MASs) model with directed topology is formulated to the optimal tracking control problem via online deep reinforcement learning method. Based on graph theory, matrix analysis, Lyapunov stability, deep learning and optimal control, the optimality of value function and the stability of the consensus error systems for the unknown second-order systems are guaranteed for each agent. The results show that the designed policy iteration algorithm not only stabilizes the distributed dynamic systems, but also makes all agents’ position and velocity states reach consensus, respectively. Finally, the correctness of our theoretical results is illustrated under two numerical simulations based on the designing model-free actor-critic networks.

Collective Motion and Self-Organization of a Swarm of UAVs: A Cluster-Based Architecture.

This study proposes a collective motion and self-organization control of a swarm of 10 UAVs, which are divided into two clusters of five agents each. A cluster is a group of UAVs in a dedicated area and multiple clusters make a swarm. This paper designs the 3D model of the whole environment by applying graph theory. To address the aforesaid issues, this paper designs a hybrid meta-heuristic algorithm by merging the particle swarm optimization (PSO) with the multi-agent system (MAS). First, PSO only provides the best agents of a cluster. Afterward, MAS helps to assign the best agent as the leader of the nth cluster. Moreover, the leader can find the optimal path for each cluster. Initially, each cluster contains agents at random positions. Later, the clusters form a formation by implementing PSO with the MAS model. This helps in coordinating the agents inside the nth cluster. However, when two clusters combine and make a swarm in a dynamic environment, MAS alone is not able to fill the communication gap of n clusters. This study does it by applying the Vicsek-based MAS connectivity and synchronization model along with dynamic leader selection ability. Moreover, this research uses a B-spline curve based on simple waypoint defined graph theory to create the flying formations of each cluster and the swarm. Lastly, this article compares the designed algorithm with the NSGA-II model to show that the proposed model has better convergence and durability, both in the individual clusters and inside the greater swarm.

Expressiveness and Nash Equilibrium in Iterated Boolean Games

We define and investigate a novel notion of expressiveness for temporal logics that is based on game theoretic equilibria of multi-agent systems. We use iterated Boolean games as our abstract model of multi-agent systems [Gutierrez et al. 2013, 2015a]. In such a game, each agent has a goal , represented using (a fragment of) Linear Temporal Logic ( ) . The goal captures agent ’s preferences, in the sense that the models of represent system behaviours that would satisfy . Each player controls a subset of Boolean variables , and at each round in the game, player is at liberty to choose values for variables in any way that she sees fit. Play continues for an infinite sequence of rounds, and so as players act they collectively trace out a model for , which for every player will either satisfy or fail to satisfy their goal. Players are assumed to act strategically, taking into account the goals of other players, in an attempt to bring about computations satisfying their goal. In this setting, we apply the standard game-theoretic concept of (pure) Nash equilibria. The (possibly empty) set of Nash equilibria of an iterated Boolean game can be understood as inducing a set of computations, each computation representing one way the system could evolve if players chose strategies that together constitute a Nash equilibrium. Such a set of equilibrium computations expresses a temporal property—which may or may not be expressible within a particular fragment. The new notion of expressiveness that we formally define and investigate is then as follows: What temporal properties are characterised by the Nash equilibria of games in which agent goals are expressed in specific fragments of ? We formally define and investigate this notion of expressiveness for a range of fragments. For example, a very natural question is the following: Suppose we have an iterated Boolean game in which every goal is represented using a particular fragment of : is it then always the case that the equilibria of the game can be characterised within ? We show that this is not true in general.

Real-time multi-agent systems: rationality, formal model, and empirical results

Since its dawn as a discipline, Artificial Intelligence (AI) has focused on mimicking the human mental processes. As AI applications matured, the interest for employing them into real-world complex systems (i.e., coupling AI with Cyber-Physical Systems—CPS) kept increasing. In the last decades, the multi-agent systems (MAS) paradigm has been among the most relevant approaches fostering the development of intelligent systems. In numerous scenarios, MAS boosted distributed autonomous reasoning and behaviors. However, many real-world applications (e.g., CPS) demand the respect of strict timing constraints. Unfortunately, current AI/MAS theories and applications only reason “about time” and are incapable of acting “in time” guaranteeing any timing predictability. This paper analyzes the MAS compliance with strict timing constraints (real-time compliance)—crucial for safety-critical applications such as healthcare, industry 4.0, and automotive. Moreover, it elicits the main reasons for the lack of real-time satisfiability in MAS (originated from current theories, standards, and implementations). In particular, traditional internal agent schedulers (general-purpose-like), communication middlewares, and negotiation protocols have been identified as co-factors inhibiting real-time compliance. To pave the road towards reliable and predictable MAS, this paper postulates a formal definition and mathematical model of real-time multi-agent systems (RT-MAS). Furthermore, this paper presents the results obtained by testing the dynamics characterizing the RT-MAS model within the simulator MAXIM-GPRT. Thus, it has been possible to analyze the deadline miss ratio between the algorithms employed in the most popular frameworks and the proposed ones. Finally, discussing the obtained results, the ongoing and future steps are outlined.

Reliability evaluation of multi-agent integrated energy systems with fully distributed communication

The reliability evaluation of integrEated energy systems becomes very important, because it is the basis of planning and operation. However, different stakeholders in integrated energy systems bring difficulty to evaluation process due to data privacy protect. In this paper, a reliability evaluation approach for multi-agent integrated energy systems via fully distributed communication is presented. Firstly, a reliability evaluation model of multi-agent integrated energy systems is established, where wind turbines, plug-in electric vehicles and gas storages are considered in the structure. Moreover, a re-dispatching model with voltage and gas pressure constraints is presented to minimize operation cost, when a contingent incident happens. Through second-order cone and Big M relaxation, the re-dispatching model is reformulated to a solvable one. Furthermore, a state assessment method with fully distributed communication is proposed to reduce information exchange, which is presented based on Monte Carlo simulation and alternating direction method of multipliers with Gaussian back substitution. Reliability indices of each agent and whole system can be calculated via a little non-privacy message transferring. Finally, the proposed model and approach are tested on a modified MA-IES. The relevant failure among different sub-systems is analyzed and the effectiveness of fully distributed communication is validated.