Self-interested Agents Research Articles

Resolving social dilemmas with minimal reward transfer

Social dilemmas present a significant challenge in multi-agent cooperation because individuals are incentivised to behave in ways that undermine socially optimal outcomes. Consequently, self-interested agents often avoid collective behaviour. In response, we formalise social dilemmas and introduce a novel metric, the general self-interest level, to quantify the disparity between individual and group rationality in such scenarios. This metric represents the maximum proportion of their individual rewards that agents can retain while ensuring that a social welfare optimum becomes a dominant strategy. Our approach diverges from traditional concepts of altruism, instead focusing on strategic reward redistribution. By transferring rewards among agents in a manner that aligns individual and group incentives, rational agents will maximise collective welfare while pursuing their own interests. We provide an algorithm to compute efficient transfer structures for an arbitrary number of agents, and introduce novel multi-player social dilemma games to illustrate the effectiveness of our method. This work provides both a descriptive tool for analysing social dilemmas and a prescriptive solution for resolving them via efficient reward transfer contracts. Applications include mechanism design, where we can assess the impact on collaborative behaviour of modifications to models of environments.

Decentralized multi-agent path finding framework and strategies based on automated negotiation

This paper introduces a negotiation framework to solve the Multi-Agent Path Finding (MAPF) Problem for self-interested agents in a decentralized fashion. The framework aims to achieve a good trade-off between the privacy of the agents and the effectiveness of solutions. Accordingly, a token-based bilateral negotiation protocol and two negotiation strategies are presented. The experimental results over four different settings of the MAPF problem show that the proposed approach could find conflict-free path solutions albeit suboptimally, especially when the search space is large and high-density. In contrast, Explicit Estimation Conflict-Based Search (EECBS) struggles to find optimal solutions. Besides, deploying a sophisticated negotiation strategy that utilizes information about local density for generating alternative paths can yield remarkably better solution performance in this negotiation framework.

Islands of cooperation emerge by stigmergic interactions in iterated spatial games.

This article focuses on the emergence of cooperation in societies of self-interested agents. In particular, it introduces a mechanism based on indirect-stigmergic-interactions between agents moving in an environment, to express the likeliness of finding cooperative partners. On the one hand, agents that find themselves cooperating with others emit pheromones in their current location, expressing the presence of agents willing to cooperate. On the other hand, agents that sense pheromones tend to move towards regions with a higher pheromone concentration. Results show that this mechanism leads to the emergence of spatial regions where cooperation can be effectively sustained, and in which agents can overall get better payoffs than those agents not taking into account pheromones in their choices.

Bounded incentives in manipulating the probabilistic serial rule

The Probabilistic Serial mechanism is valued for its fairness and efficiency in addressing the random assignment problem. However, it lacks truthfulness, meaning it works well only when agents' stated preferences match their true ones. Significant utility gains from strategic actions may lead self-interested agents to manipulate the mechanism, undermining its practical adoption. To gauge the potential for manipulation, we explore an extreme scenario where a manipulator has complete knowledge of other agents' reports and unlimited computational resources to find their best strategy. We establish tight incentive ratio bounds of the mechanism. Furthermore, we complement these worst-case guarantees by conducting experiments to assess an agent's average utility gain through manipulation. The findings reveal that the incentive for manipulation is very small. These results offer insights into the mechanism's resilience against strategic manipulation, moving beyond the recognition of its lack of incentive compatibility.

A new bargaining solution for finite offer spaces

AbstractThe bargaining problem deals with the question of how far a negotiating agent should concede to its opponent. Classical solutions to this problem, such as the Nash bargaining solution (NBS), are based on the assumption that the set of possible negotiation outcomes forms a continuous space. Recently, however, we proposed a new solution to this problem for scenarios with finite offer spaces de Jonge and Zhang (Auton Agents Multi-Agent Syst 34(1):1–41, 2020). Our idea was to model the bargaining problem as a normal-form game, which we called the concession game, and then pick one of its Nash equilibria as the solution. Unfortunately, however, this game in general has multiple Nash equilibria and it was not clear which of them should be picked. In this paper we fill this gap by defining a new solution to the general problem of how to choose between multiple Nash equilibria, for arbitrary 2-player normal-form games. This solution is based on the assumption that the agent will play either ‘side’ of the game (i.e. as row-player or as column-player) equally often, or with equal probability. We then apply this to the concession game, which ties up the loose ends of our previous work and results in a proper, well-defined, solution to the bargaining problem. The striking conclusion, is that for rational and purely self-interested agents, in most cases the optimal strategy is to agree to the deal that maximizes the sum of the agents’ utilities and not the product of their utilities as the NBS prescribes.

Facility location games with ordinal preferences

In this paper we study a novel model of facility location games with ordinal preferences. There is a set of self-interested agents and a set of heterogeneous facilities. Each agent is located on a line and has an ordinal preference over the facilities. Our goal is to design strategyproof mechanisms that elicit true information (preferences and/or locations) from the agents and locate the facilities to minimize both maximum and total cost objectives as well as to maximize both minimum and total utility objectives. For the four possible objectives, we consider the 2-facility settings in which only preferences are private, or locations are private. For each possible combination of the objectives and settings, we provide lower and upper bounds on the approximation ratios of strategyproof mechanisms, which are asymptotically tight up to a constant. Furthermore, we extend some of the results to the multiple-facility setting. Finally, we discuss the generalization of our results when the agents can misreport both locations and preferences, and the case when the approximations are defined additively.

Exploration and Incentives in Reinforcement Learning

How do you incentivize self-interested agents to explore when they prefer to exploit? We consider complex exploration problems, where each agent faces the same (but unknown) Markov decision process (MDP). In contrast with traditional formulations of reinforcement learning (RL), agents control the choice of policies, whereas an algorithm can only issue recommendations. However, the algorithm controls the flow of information, and can incentivize the agents to explore via information asymmetry. We design an algorithm which explores all reachable states in the MDP. We achieve provable guarantees similar to those for incentivizing exploration in static, stateless exploration problems studied previously. From the RL perspective, we design RL mechanisms, that is, RL algorithms that interact with self-interested agents and are compatible with their incentives. This is the first paper on RL mechanisms, that is, the first paper on any scenario that combines RL and incentives, to the best of our knowledge.

Civic engagement, the leverage effect and the accountable state

A classic solution to the problem of public goods (PG) is their provision through a strong state with the power to collect taxes and to mete out penalties for non-compliance. The need for voluntary collective action remains, however, because binding the state to citizen's interests requires the latter's civic engagement. As a public good in its own right, economic theory expects civic engagement to be underprovided. We conduct the first laboratory experiment in which participants can create a socially efficient central sanctioning scheme (representing the accountable state) through a prior stage of voluntary costly actions that are theoretically ruled out for strictly self-interested agents—a social dilemma. Our experimental subjects sustain civic engagement when its cost is modest, suggesting sustainable cooperation in linked social dilemmas perhaps due to a cost-benefit calculus we call “leverage.”

Best-response planning for urban fleet coordination

The modeling of fleet vehicles as self-interested agents brings a realistic perspective to open fleet transportation research. This feature allows us to model the fleet operation from a non-cooperative point of view. In this work, we study parcel delivery in a city with limited resources (roads and charging stations). We designed and implemented a system composed of a multi-agent planner and a game-theoretic coordination algorithm: a Best-Response Fleet Planner. The system allows for the self-organization of the transportation system by coordinating a fleet of self-interested electric vehicles. The system’s operation is optimized together with resource usage while preserving the agents’ private interests, allowing each agent to plan its actions. The results show that our system has higher scalability than similar approaches, allowing it to function for a considerable number of agents in settings that feature congestion and conflicts. Additionally, overall solution quality is improved compared to other coordination systems, reducing congestion and avoiding unnecessary waiting times.

A Self-Contained Karma Economy for the Dynamic Allocation of Common Resources

This paper presents karma mechanisms, a novel approach to the repeated allocation of a scarce resource among competing agents over an infinite time. Examples include deciding which ride hailing trip requests to serve during peak demand, granting the right of way in intersections or lane mergers, or admitting internet content to a regulated fast channel. We study a simplified yet insightful formulation of these problems where at every instant two agents from a large population get randomly matched to compete over the resource. The intuitive interpretation of a karma mechanism is “If I give in now, I will be rewarded in the future.” Agents compete in an auction-like setting where they bid units of karma, which circulates directly among them and is self-contained in the system. We demonstrate that this allows a society of self-interested agents to achieve high levels of efficiency without resorting to a (possibly problematic) monetary pricing of the resource. We model karma mechanisms as dynamic population games and guarantee the existence of a stationary Nash equilibrium. We then analyze the performance at the stationary Nash equilibrium numerically. For the case of homogeneous agents, we compare different mechanism design choices, showing that it is possible to achieve an efficient and ex-post fair allocation when the agents are future aware. Finally, we test the robustness against agent heterogeneity and propose remedies to some of the observed phenomena via karma redistribution.

Distributed energy trading on networked energy hubs under network constraints

A distributed energy trading scheme with non-discriminatory pricing for a cluster of networked energy hubs (NEHs) is proposed. First, each energy hub (EH) is treated as a self-interested agent. The hybrid AC/DC microgrid (MG)-embedded EH model is proposed to optimize the operating costs under corresponding local energy balance constraints. The supply limits of the input energy systems, e.g., electrical feeders and natural gas pipelines, are represented as the global coupling constraints among NEHs. Then, to obtain the optimal operation and trading strategies, the distributed energy trading is formulated as a generalized Nash game (GNG). To ensure the solubility of the GNG problem, the existence and uniqueness of the generalized Nash equilibrium (GNE) are proved. Furthermore, to transform the complexity of the solution, the multivariable GNG problem is reformulated as a N+1 Nash game (NG) without coupling constraints, the equivalence between NG and the solution set of variational inequality (VI) problem is established. Then, an efficient distributed projection-based algorithm is proposed to compute a Nash equilibrium (NE) for the NG problem. Finally, a potential game-based centralized solution method is also implemented as a baseline, and the comparison of simulation results demonstrates the effectiveness of our proposed algorithm.

Prospection and delay of gratification support the development of calculated reciprocity

Humans frequently benefit others strategically to elicit future cooperation. While such forms of calculated reciprocity are powerful in eliciting cooperative behaviors even among self-interested agents, they depend on advanced cognitive and behavioral capacities such as prospection (representing and planning for future events) and extended delay of gratification. In fact, it has been proposed that these constraints help explain why calculated reciprocity exists in humans and is rare or even absent in other animals. The current study investigated the cognitive foundation of calculated reciprocity by examining its ontogenetic emergence in relation to key aspects of children's cognitive development. Three-to-five-year-old children from the US (N = 72, mostly White, from mixed socioeconomic backgrounds) first completed a cognitive test battery assessing the cognitive capacities hypothesized to be foundational for calculated reciprocity. In a second session, children participated in a calculated reciprocity task in which they could decide how many resources to share with a partner who later had the opportunity to reciprocate (reciprocity condition) and with a partner who could not reciprocate (control condition). Results indicated a steep developmental emergence of calculated reciprocity between 3 and 5 years of age. Further analyses showed that measures of delay of gratification and prospection were important predictors of children's rate of calculated reciprocity, even when controlling for age and after including a measure of verbal ability. By contrast, theory of mind abilities were unrelated to children's reciprocal behavior. This is the first systematic investigation of essential cognitive capacities for calculated reciprocity. We discuss prospection and delay of gratification as two domain-general capacities that are utilized for calculated reciprocity and which could explain developmental as well as species-differences in cooperation.

Rational Altruism

This article proposes a game-theoretic equilibrium concept promoting endogenous cooperation through self-interested agents optimally choosing how much they care about their opponents’ welfare. Rationally altruistic equilibria always exist in finite games as long as a publicly observed signal is available. Characterizations of the equilibrium concept highlight that it is most likely to have a bite when players’ interests are sufficiently dissimilar but not diametrically opposed to each other—such as in the prisoner’s dilemma. In the presence of more than two players, rational altruism can take an opportunistic form and serve as a device of collusion through implicit coalition formation.

Coordinating Multi-party Vehicle Routing with Location Congestion via Iterative Best Response

This work is motivated by a real-world problem of coordinating B2B pickup-delivery operations to shopping malls involving multiple non-collaborative logistics service providers (LSPs) in a congested city where space is scarce. This problem can be categorized as a vehicle routing problem with pickup and delivery, time windows and location congestion with multiple LSPs (or ML-VRPLC in short), and we propose a scalable, decentralized, coordinated planning approach via iterative best response. We formulate the problem as a strategic game where each LSP is a self-interested agent but is willing to participate in a coordinated planning as long as there are sufficient incentives. Through an iterative best response procedure, agents adjust their schedules until no further improvement can be obtained to the resulting joint schedule. We seek to find the best joint schedule which maximizes the minimum gain achieved by any one LSP, as LSPs are interested in how much benefit they can gain rather than achieving a system optimality. We compare our approach to a centralized planning approach and our experiment results show that our approach is more scalable and is able to achieve on average 10% more gain within an operationally realistic time limit.

A real-time balancing market optimization with personalized prices: From bilevel to convex

This paper studies the static economic optimization problem of a system with a single aggregator and multiple prosumers in a Real-Time Balancing Market (RTBM). The aggregator, as the agent responsible for portfolio balancing, needs to minimize the cost for imbalance satisfaction in real-time by proposing a set of optimal personalized prices to the prosumers. On the other hand, the prosumers, as price taker and self-interested agents, want to maximize their profit by changing their supplies or demands and providing flexibility based on the proposed personalized prices. We model this problem as a bilevel optimization problem. We first show that the optimal solution of this bilevel optimization problem can be found by solving an equivalent convex problem. In contrast to the state-of-the-art Mixed-Integer Programming (MIP)-based approach to solve bilevel problems, this convex equivalent has very low computation time and is appropriate for real-time applications. Next, we compare the optimal solutions of the proposed personalized scheme and a uniform pricing scheme. We prove that, under the personalized pricing scheme, more prosumers contribute to the RTBM and the aggregator’s cost is less. Finally, we verify the analytical results of this work by means of numerical case studies and simulations.

Selective Exploration Algorithm for Coalition Formation

Coalition formation with task dependencies has been the focus of few of the literature on multi-agent coalition formation. In contrast, very little attention has been given to the case where each agent has several alternative sets of tasks leading it to its goal satisfaction. The task dependencies impact the feasibility of the coalitions depending on the considered tasks alternative at a given time. Hence, the performance of a single task, whether by coalition formation (group of agents) or by a single performance, could affect the feasibility of other coalitions in the system. However, these task dependencies play a crucial role in many real-world multi-agent applications. Against this background, we consider in this paper multiple self-interested agents each of which has a goal it needs to achieve by performing a set of tasks. Each agent has several alternative sets of tasks leading it to its goal satisfaction. The tasks in an alternative exhibit dependencies and require sequential execution. So, to jointly achieve goals, the agents may form interdependent coalitions. We introduce a new algorithm that we call the Selective Exploration Algorithm (SEA) for coalition formation that accounts for the task dependencies to consider only feasible coalitions and reduce the size of the search space to explore and identify the optimal coalition structure.

Two Strongly Truthful Mechanisms for Three Heterogeneous Agents Answering One Question

Peer prediction mechanisms incentivize self-interested agents to truthfully report their signals even in the absence of verification by comparing agents’ reports with their peers. We propose two new mechanisms, Source and Target Differential Peer Prediction, and prove very strong guarantees for a very general setting. Our Differential Peer Prediction mechanisms are strongly truthful : Truth-telling is a strict Bayesian Nash equilibrium. Also, truth-telling pays strictly higher than any other equilibria, excluding permutation equilibria, which pays the same amount as truth-telling. The guarantees hold for asymmetric priors among agents, which the mechanisms need not know ( detail-free ) in the single question setting . Moreover, they only require three agents , each of which submits a single item report : two report their signals (answers), and the other reports her forecast (prediction of one of the other agent’s reports). Our proof technique is straightforward, conceptually motivated, and turns on the logarithmic scoring rule’s special properties. Moreover, we can recast the Bayesian Truth Serum mechanism [ 20 ] into our framework. We can also extend our results to the setting of continuous signals with a slightly weaker guarantee on the optimality of the truthful equilibrium.

A regulatory arbitrage game: Off-balance-sheet leverage and financial fragility

This study examines a simple banking system in a game-theoretic framework wherein banks act as self-interested agents to maximize leverage at the expense of overall financial stability. The resultant strategic inefficiency raises concerns about how banks manage the "financial stability" good, which is appropriated into a "tragedy of the commons." We conceptualize the inefficiency using the price of anarchy introduced by Koutsoupias & Papadimitriou (2009). We seek the optimal regulatory framework that minimizes the price of anarchy or the degree of financial fragility.

The Price of Incentivizing Exploration: A Characterization via Thompson Sampling and Sample Complexity

We consider “incentivized exploration”: a version of multiarmed bandits where the choice of arms is controlled by self-interested agents. The algorithm can only issue recommendations and needs to incentivize the agents to explore, even though they prefer to exploit. The algorithm controls the flow of information and relies on information asymmetry to create incentives. This line of work, motivated by misaligned incentives in recommendation systems, has received considerable attention in the “economics and computation” community. We focus on the “price of incentives”: the loss in performance, broadly construed, incurred for the sake of incentive compatibility. We prove that Thompson sampling, a standard bandit algorithm, is incentive compatible if initialized with sufficiently many data points. The performance loss because of incentives is, therefore, limited to the initial rounds when these data points are collected. The problem is largely reduced to that of sample complexity. How many rounds are needed to collect even one sample of each arm? We zoom in on this question, which is perhaps the most basic question one could ask about incentivized exploration. We characterize the dependence on agents' beliefs and the number of arms (which was essentially ignored in prior work), providing matching upper and lower bounds. Typically, the optimal sample complexity is polynomial in the number of arms and exponential in the “strength of beliefs.”

The Black Box as a Control for Payoff-Based Learning in Economic Games.

The black box method was developed as an "asocial control" to allow for payoff-based learning while eliminating social responses in repeated public goods games. Players are told they must decide how many virtual coins they want to input into a virtual black box that will provide uncertain returns. However, in truth, they are playing with each other in a repeated social game. By "black boxing" the game's social aspects and payoff structure, the method creates a population of self-interested but ignorant or confused individuals that must learn the game's payoffs. This low-information environment, stripped of social concerns, provides an alternative, empirically derived null hypothesis for testing social behaviours, as opposed to the theoretical predictions of rational self-interested agents (Homo economicus). However, a potential problem is that participants can unwittingly affect the learning of other participants. Here, we test a solution to this problem in a range of public goods games by making participants interact, unknowingly, with simulated players ("computerised black box"). We find no significant differences in rates of learning between the original and the computerised black box, therefore either method can be used to investigate learning in games. These results, along with the fact that simulated agents can be programmed to behave in different ways, mean that the computerised black box has great potential for complementing studies of how individuals and groups learn under different environments in social dilemmas.