Homogeneous Agents Research Articles

Information consistency and inhibition impact on diffusion in networks

This paper introduces the Principal-Agent mechanism to study the diffusion of information in complex networks. Hot list information is defined as principal information, information that is homogeneous with hot list information is defined as homogeneous agent information, and information that is heterogeneous with hot list information is defined as heterogeneous agent information. The theory of diffusion dynamics is used to study the phenomenon that homogeneous information promotes each other and heterogeneous information inhibits each other in the process of competitive diffusion of network information. Based on the traditional SIR model, this paper introduces the Principal-Agent mechanism, establishes the PA_SIR diffusion model and studies the mutual influence of information in the process of competitive diffusion of multiple information. The research shows that in the process of competitive diffusion of multiple information, principal information has a positive incentive effect on homogeneous agent information through people’s interest orientation, herd mentality and topic public opinion and has a negative incentive effect on heterogeneous agent information. Therefore, if you want your published information to attract more attention, the best release time is when homogeneous information is on the hot search list, which will help to attract people’s attention more quickly.

Integrate-and-fire model of disease transmission.

We create an epidemiological susceptible-infected-susceptible model of disease transmission using integrate-and-fire nodes on a network, allowing memory of previous interactions and infections. Agents in the network sum infectious matter from their nearest neighbors at every time step, until they exceed their infection threshold, at which point they "fire" and become infected for as long as the recovery time. The model has memory of previous interactions by tracking the amount of infectious matter carried by agents as well as just binary infected or susceptible states, and the model has memory of previous infections by modeling immunity as increasing the infection threshold after recovery. Creating a simulation of the model on networks with a power-law degree distribution and homogeneous agent parameters, we find a single strain version of the model matches well with the England COVID-19 case data, with a root-mean-squared error of 0.014%. A simulation of a multistrain version of the model (where there is cross-strain immunity) matches well with the influenza strain A and strain B case numbers in Canada, with a root-mean-squared error of 0.002% and 0.0012%, respectively, though due to the coupling in the model, both strains peak in phase. Since the dynamics of the model successfully capture real-life transmission dynamics, we test interventions to study their effect on case numbers, with both quarantining and social gathering restrictions lowering the peak. Since the model has memory, the stricter the intervention, the higher the secondary peak when the restriction is removed, showing that interventions change only the shape of the curves and not the overall number infected in the population.

Distributed entropy-regularized multi-agent reinforcement learning with policy consensus

Sample efficiency is a limiting factor for existing distributed multi-agent reinforcement learning (MARL) algorithms over networked multi-agent systems. In this paper, the sample efficiency problem is tackled by formally incorporating the entropy regularization into the distributed MARL algorithm design. Firstly, a new entropy-regularized MARL problem is formulated under the model of networked multi-agent Markov decision processes with observation-based policies and homogeneous agents, where the policy parameter sharing among the agents provably preserves the optimality. Secondly, an on-policy distributed actor–critic algorithm is proposed, where each agent shares its parameters of both the critic and actor for consensus update. Then, the convergence analysis of the proposed algorithm is provided based on the stochastic approximation theory under the assumption of linear function approximation of the critic. Furthermore, a practical off-policy version of the proposed algorithm is developed which possesses scalability, data efficiency and learning stability. Finally, the proposed distributed algorithm is compared against the solid baselines including two classic centralized training algorithms in the multi-agent particle environment, whose learning performance is empirically demonstrated through extensive simulation experiments.

The role of institutions in shaping the growth-aid relationship

Empirical evidence on the relationship between aid and economic growth is mixed and inconclusive. This paper proposes a theory to explain these contradictory findings. We build an endogenous growth model with a productive public good and homogeneous agents who allocate their time to both work and the appropriation of public resources. Aid increases public resources, raising the provision of the productive public good, but promotes rent-seeking. As recent empirical evidence suggests, a hump-shaped relationship between aid and growth emerges: too much aid is counterproductive for growth, particularly when institutions are weak. Aid transmits growth from the donor to the recipient country but harms income convergence and even prevents convergence among ex-ante identical countries when aid exceeds a certain threshold. Institutional improvements raise such a threshold. Thus, countries with lower income and lower institutional quality should receive less aid, unless an institutional reform is taken as a previous step to receive that aid.

I-Rebalance: Personalized Vehicle Repositioning for Supply Demand Balance

Ride-hailing platforms have been facing the challenge of balancing demand and supply. Existing vehicle reposition techniques often treat drivers as homogeneous agents and relocate them deterministically, assuming compliance with the reposition. In this paper, we consider a more realistic and driver-centric scenario where drivers have unique cruising preferences and can decide whether to take the recommendation or not on their own. We propose i-Rebalance, a personalized vehicle reposition technique with deep reinforcement learning (DRL). i-Rebalance estimates drivers' decisions on accepting reposition recommendations through an on-field user study involving 99 real drivers. To optimize supply-demand balance and enhance preference satisfaction simultaneously, i-Rebalance has a sequential reposition strategy with dual DRL agents: Grid Agent to determine the reposition order of idle vehicles, and Vehicle Agent to provide personalized recommendations to each vehicle in the pre-defined order. This sequential learning strategy facilitates more effective policy training within a smaller action space compared to traditional joint-action methods. Evaluation of real-world trajectory data shows that i-Rebalance improves driver acceptance rate by 38.07% and total driver income by 9.97%.

Capital exploitation and environmental awareness: How they affect the economy and the environment in a dynamic framework

We present a model in which capital and environmental quality co-evolve over time. To improve the environmental quality, the government intervenes by means of a limitation of the capital use and awareness campaigns. In case of severe degradation of the environment, a restriction on capital use is introduced that is proportional to the damage caused by human activity; at the same time, awareness campaigns are used to increase the public concern about sustainability. By means of a discrete-time dynamical system and considering homogeneous agents, we found that multiple equilibria may exist and that awareness campaigns are a useful tool to push an economy toward sustainable levels of production. The limitation in the use of available capital, however, might be useless or even harmful, deteriorating the level of capital disposable for those countries that are trapped in an equilibrium in which the environmental quality is low.

Phase coexistence in the fully heterogeneous Hegselmann-Krause opinion dynamics model.

We present an extensive study of the joint effects of heterogeneous social agents and their heterogeneous social links in a bounded confidence opinion dynamics model. The full phase diagram of the model is explored for two different network's topologies and compared to two opposed extreme cases: on one hand, the heterogeneous agents constitute a mixed population and on the other, their interactions are modeled by a lattice. The results show that when agents prone to compromise coexist with close-minded ones, the steady state of the dynamics shows coexistent phases. In particular, unlike the case of homogeneous agents in networks, or heterogeneous agents in a fully mixed population, it is possible that the society ends up in consensus around one extreme opinion. Moreover, during the dynamics, the consensus may be overturned from one extreme to the other of the opinion space. We also show that the standard order parameter, the normalized average size of the largest opinion cluster, may be misleading in this case, as it hides the existence of these phases. The phase where the opinion of the society is overturned does not require the presence of agents with special characteristics, (stubborn, extremists, etc.); it results from the interplay of agents which have agreed on an extreme opinion with the remaining group that holds the opposite one. Among the former, some may be prone to compromise with other agents which are out of the majority group, these agents, according to their location in the network, may act like bridges between the two groups and slowly attract the whole society to the other extreme.

Suboptimal consensus protocol design for a class of multiagent systems

This article presents a new technique for suboptimal consensus protocol design for a class of multiagent systems. The proposed technique is based upon the extension of newly developed sufficient conditions for suboptimal linear–quadratic optimal control design, which are derived in this paper by an explication of a noniterative solution technique of the infinite-horizon linear quadratic regulation problem in Krotov framework. For suboptimal consensus protocol design, the structural requirements on the overall feedback gain matrix, which are inherently imposed by agents dynamics and their interaction topology, are recast on a specific matrix introduced in a suitably formulated convex optimization problem. As a result, preassigning the identical feedback gain matrices to a network of homogeneous agents, which act on the relative state variables with respect to their neighbors is not required. The suboptimality of the computed control laws is quantified by implicitly deriving an upper bound on the cost in terms of the solution of a convex optimization problem and initial conditions of the agents instead of specifying it a priori. Numerical examples are provided to demonstrate the implementation of proposed approaches and their comparison with existing methods in the literature.

Multiagent Reinforcement Learning With Heterogeneous Graph Attention Network.

Most recent research on multiagent reinforcement learning (MARL) has explored how to deploy cooperative policies for homogeneous agents. However, realistic multiagent environments may contain heterogeneous agents that have different attributes or tasks. The heterogeneity of the agents and the diversity of relationships cause the learning of policy excessively tough. To tackle this difficulty, we present a novel method that employs a heterogeneous graph attention network to model the relationships between heterogeneous agents. The proposed method can generate an integrated feature representation for each agent by hierarchically aggregating latent feature information of neighbor agents, with the importance of the agent level and the relationship level being entirely considered. The method is agnostic to specific MARL methods and can be flexibly integrated with diverse value decomposition methods. We conduct experiments in predator-prey and StarCraft Multiagent Challenge (SMAC) environments, and the empirical results demonstrate that the performance of our method is superior to existing methods in several heterogeneous scenarios.

Graphon mean-field control for cooperative multi-agent reinforcement learning

The marriage between mean-field theory and reinforcement learning has shown a great capacity to solve large-scale control problems with homogeneous agents. To break the homogeneity restriction of mean-field theory, a recent interest is to introduce graphon theory to the mean-field paradigm. In this paper, we propose a graphon mean-field control (GMFC) framework to approximate cooperative heterogeneous multi-agent reinforcement learning (MARL) with nonuniform interactions and heterogeneous reward functions and state transition functions among agents and show that the approximate order is of O(1N), with N the number of agents. By discretizing the graphon index of GMFC, we further introduce a smaller class of GMFC called block GMFC, which is shown to well approximate cooperative MARL in terms of the value function and the policy. Finally, we design a Proximal Policy Optimization based algorithm for block GMFC that converges to the optimal policy of cooperative MARL. Our empirical studies on several examples demonstrate that our GMFC approach is comparable with the state-of-art MARL algorithms while enjoying better scalability.

Toward multi-target self-organizing pursuit in a partially observable Markov game

The multiple-target self-organizing pursuit (SOP) problem has wide applications and has been considered a challenging self-organization game for distributed systems, in which intelligent agents cooperatively pursue multiple dynamic targets with partial observations. This work proposes a framework for decentralized multi-agent systems to improve the implicit coordination capabilities in search and pursuit. We model a self-organizing system as a partially observable Markov game (POMG) featured by large-scale, decentralization, partial observation, and noncommunication. The proposed distributed algorithm–fuzzy self-organizing cooperative coevolution (FSC2) is then leveraged to resolve the three challenges in multi-target SOP: distributed self-organizing search (SOS), distributed task allocation, and distributed single-target pursuit. FSC2 includes a coordinated multi-agent deep reinforcement learning (MARL) method that enables homogeneous agents to learn natural SOS patterns. Additionally, we propose a fuzzy-based distributed task allocation method, which locally decomposes multi-target SOP into several single-target pursuit problems. The cooperative coevolution principle is employed to coordinate distributed pursuers for each single-target pursuit problem. Therefore, the uncertainties of inherent partial observation and distributed decision-making in the POMG can be alleviated. The experimental results demonstrate that by decomposing the SOP task, FSC2 achieves superior performance compared with other implicit coordination policies fully trained by general MARL algorithms. The scalability of FSC2 is proved that up to 2048 FSC2 agents perform efficient multi-target SOP with almost 100% capture rates. Empirical analyses and ablation studies verify the interpretability, rationality, and effectiveness of component algorithms in FSC2.

Budget-feasible mechanisms for proportionally selecting agents from groups

In many social domains involving collective decision-making (e.g., committee selection and survey sampling), it is often desirable to select individuals from different population groups to achieve proportional representation (e.g., to represent the opinions of each group). For instance, in the selection of a committee (e.g., to form a working group within a company), the planner would like to select agents from different groups to represent their respective groups proportionally. Typically, there are intrinsic private costs for agents to represent their groups, and the planner would like to compensate the selected agents via some form of payments, which is constrained by the planner's available budget. As the costs are unknown to the planner, the planner is required to design incentive mechanisms to elicit agents' real costs and provide payments (or monetary incentives) to the selected agents to ensure proportional representation and the total payments do not exceed the budget. Such a mechanism design setting falls into the budget-feasible mechanism design paradigm. However, existing budget-feasible mechanisms only consider all agents to be in the same group with non-proportional objectives. To study the above-mentioned setting, we consider the problem of designing budget-feasible mechanisms for selecting agents with private costs from various groups to ensure proportional representation, where the minimum proportion of the overall value of the selected agents from each group is maximized. We study this problem by first considering the setting with homogeneous agents who have identical values to the planner. Depending on agents' membership in the groups, we consider two models: a single group model where each agent belongs to only one group, and a multiple group model where each agent may belong to multiple groups. We propose novel budget-feasible proportion-representative mechanisms for these models that require different selection methods, i.e., a novel greedy mechanism that considers all possible proportion ratios for the single group model and a novel mechanism that leverages the Max-Flow algorithm to evaluate the proportional representation for the multiple group model, to choose representative agents from each group. The proposed mechanisms guarantee theoretical properties of individual rationality, budget-feasibility, truthfulness, and approximation performance on maximizing the minimum proportional representation of each group. We also provide a matching lower bound for budget-feasible proportion-representative mechanisms. Finally, we non-trivially extend these mechanisms to the settings of heterogeneous agents who can have different values to the planner under the two models.

Controllability of consensus of multi-agent networks over heterogeneous dynamics

This work investigates controllability issues under the leader–follower framework of consensus problem on multi-agent networks where its agents exhibit the heterogeneous dynamics involving first-, second-, and third-order integrator agents on a continuous time-scale. The communication topology is directed, time-invariant, and the reliability of all interactions are unity. We employ two kinds of consensus protocols that are linear and distributed. The necessary and sufficient criteria for the controllability of followers’ dynamics through the selected leaders are obtained in terms of matrix-rank conditions. We also obtain some necessary conditions of controllability involving eigenvectors of system matrices. It is verified that, if either of the first-, second-, or third-order integrator agents are absent, then the controllability of the network becomes equivalent to the controllability of the communication topology just like those of the networks with homogeneous dynamic agents. Obtained results are verified with illustrative examples.

On the internal stability of diffusively coupled multi-agent systems and the dangers of cancel culture

We study internal stability in the context of diffusively-coupled control architectures, common in multi-agent systems (i.e. the celebrated consensus protocol), for linear time-invariant agents. We derive a condition under which the system cannot be stabilized by any controller from that class. In the finite-dimensional case the condition states that diffusive controllers cannot stabilize agents that share common unstable dynamics, directions included. This class always contains the group of homogeneous unstable agents, like integrators. We argue that the underlying reason is intrinsic cancellations of unstable agent dynamics by such controllers, even static ones, where directional properties play a key role. The intrinsic lack of internal stability explains the notorious behavior of some distributed control protocols when affected by measurement noise or exogenous disturbances.

A semi-independent policies training method with shared representation for heterogeneous multi-agents reinforcement learning.

Humans do not learn everything from the scratch but can connect and associate the upcoming information with the exchanged experience and known knowledge. Such an idea can be extended to cooperated multi-reinforcement learning and has achieved its success on homogeneous agents by means of parameter sharing. However, it is difficult to straightforwardly apply parameter sharing when dealing with heterogeneous agents thanks to their individual forms of input/output and their diverse functions and targets. Neuroscience has provided evidence that our brain creates several levels of experience and knowledge-sharing mechanisms that not only exchange similar experiences but also allow for sharing of abstract concepts to handle unfamiliar situations that others have already encountered. Inspired by such a brain's functions, we propose a semi-independent training policy method that can well tackle the conflict between parameter sharing and specialized training for heterogeneous agents. It employs a shared common representation for both observation and action, enabling the integration of various input and output sources. Additionally, a shared latent space is utilized to maintain a balanced relationship between the upstream policy and downstream functions, benefiting each individual agent's target. From the experiments, it can approve that our proposed method outperforms the current mainstream algorithms, especially when handling heterogeneous agents. Empirically, our proposed method can also be improved as a more general and fundamental heterogeneous agents' reinforcement learning structure for curriculum learning and representation transfer. All our code is open and released on https://gitlab.com/reinforcement/ntype.

Super‐stable homogeneous embolic agents advance the treatment of hepatocellular carcinoma

Super‐stable homogeneous embolic agents advance the treatment of hepatocellular carcinoma

Practical applications of Network Data Envelopment Analysis

Subject. The article considers a methodology for assessing the comparative effectiveness of the activity of homogeneous economic agents, i.e. Data Envelopment Analysis. Objectives. The focus is on systematization and classification of modern practical applications of network Data Envelopment Analysis, identification of types of additional information that can be extracted from solving problems of network DEA for the strategic management of companies/organizations. Methods. The study rests on systematic literature review. Results. At present, multi-stage DEA models are most actively used to model and evaluate the performance of banks, supply chains consisting of a “supplier-manufacturer-distributor” link, innovative and high-tech companies (or territories), and companies whose activities are regulated by strict environmental standards. Least of all, multi-stage DEA models are so far used to model consumer behavior as a sequential process consisting of many stages, which is explained by the underdevelopment of approaches to measuring consumer behavior factors. Conclusions. The main difference between the types of multi-stage network models is the absence or presence of common inputs for several stages, which are divided in a certain proportion between the stages (subsystems). This factor significantly affects the type of optimization model and approaches to its solution. The presence of common inputs gives rise to the need to solve an additional optimization problem for the distribution of resources between subsystems.

A Self-Contained Karma Economy for the Dynamic Allocation of Common Resources

This paper presents karma mechanisms, a novel approach to the repeated allocation of a scarce resource among competing agents over an infinite time. Examples include deciding which ride hailing trip requests to serve during peak demand, granting the right of way in intersections or lane mergers, or admitting internet content to a regulated fast channel. We study a simplified yet insightful formulation of these problems where at every instant two agents from a large population get randomly matched to compete over the resource. The intuitive interpretation of a karma mechanism is “If I give in now, I will be rewarded in the future.” Agents compete in an auction-like setting where they bid units of karma, which circulates directly among them and is self-contained in the system. We demonstrate that this allows a society of self-interested agents to achieve high levels of efficiency without resorting to a (possibly problematic) monetary pricing of the resource. We model karma mechanisms as dynamic population games and guarantee the existence of a stationary Nash equilibrium. We then analyze the performance at the stationary Nash equilibrium numerically. For the case of homogeneous agents, we compare different mechanism design choices, showing that it is possible to achieve an efficient and ex-post fair allocation when the agents are future aware. Finally, we test the robustness against agent heterogeneity and propose remedies to some of the observed phenomena via karma redistribution.

Hierarchical Attention Master–Slave for heterogeneous multi-agent reinforcement learning

Most multi-agent reinforcement learning (MARL) approaches optimize strategy by improving itself, while ignoring the limitations of homogeneous agents that may have single function. However, in reality, the complex tasks tend to coordinate various types of agents and leverage advantages from one another. Therefore, it is a vital research issue how to establish appropriate communication among them and optimize decision. To this end, we propose a Hierarchical Attention Master–Slave (HAMS) MARL, where the Hierarchical Attention balances the weight allocation within and among clusters, and the Master–Slave architecture endows agents independent reasoning and individual guidance. By the offered design, information fusion, especially among clusters, is implemented effectively, and excessive communication is avoided, moreover, selective composed action optimizes decision. We evaluate the HAMS on both small and large scale heterogeneous StarCraft II micromanagement tasks. The proposed algorithm achieves the exceptional performance with more than 80% win rates in all evaluation scenarios, which obtains an impressive win rate of over 90% in the largest map. The experiments demonstrate a maximum improvement in win rate of 47% over the best known algorithm. The results show that our proposal outperforms recent state-of-the-art approaches, which provides a novel idea for heterogeneous multi-agent policy optimization.

Lateral Transfer Learning for Multiagent Reinforcement Learning

Some researchers have introduced transfer learning mechanisms to multiagent reinforcement learning (MARL). However, the existing works devoted to cross-task transfer for multiagent systems were designed just for homogeneous agents or similar domains. This work proposes an all-purpose cross-transfer method, called multiagent lateral transfer (MALT), assisting MARL with alleviating the training burden. We discuss several challenges in developing an all-purpose multiagent cross-task transfer learning method and provide a feasible way of reusing knowledge for MARL. In the developed method, we take features as the transfer object rather than policies or experiences, inspired by the progressive network. To achieve more efficient transfer, we assign pretrained policy networks for agents based on clustering, while an attention module is introduced to enhance the transfer framework. The proposed method has no strict requirements for the source task and target task. Compared with the existing works, our method can transfer knowledge among heterogeneous agents and also avoid negative transfer in the case of fully different tasks. As far as we know, this article is the first work denoted to all-purpose cross-task transfer for MARL. Several experiments in various scenarios have been conducted to compare the performance of the proposed method with baselines. The results demonstrate that the method is sufficiently flexible for most settings, including cooperative, competitive, homogeneous, and heterogeneous configurations.