Multi-agent Setting Research Articles

Taming Strategy Logic: Non-Recurrent Fragments

Strategy Logic (▪ for short) is one of the prominent languages for reasoning about the strategic abilities of agents in a multi-agent setting. This logic extends ▪ with first-order quantifiers over the agent strategies and encompasses other formalisms, such as ▪ and ▪. The model-checking problem for ▪ and several of its fragments has been extensively studied. On the other hand, the picture is much less clear on the satisfiability front, where the problem is undecidable for the full logic. In this work, we study two fragments of One-Goal▪, where the nesting of sentences within temporal operators is constrained. We show that the satisfiability problem for these two logics, and for the corresponding fragments of ▪ and ▪, is in ▪ and ▪, respectively.

Integrating professional identities: an ethnographic study of psychoanalytic child psychotherapy in a children’s social care setting

ABSTRACT The importance of child and adolescent mental health services (CAMHS) collaborating with other agencies is of paramount concern for children with complex difficulties, including children in care. However, there is a lack of research exploring the role of child and adolescent psychoanalytic psychotherapists in multi-disciplinary and multi-agency work surrounding these children. Aims: to develop a theory of how child and adolescent psychoanalytic psychotherapists function within a social care setting, including how they position themselves in a multi-disciplinary and multi-agency environment. Methods: an ethnographic case study design, consisting of participant observation supplemented by interviews. Findings: grounded theory analysis identified that the child psychotherapists balanced three elements of their professional identity: discipline-specific identity; CAMHS team member identity; child’s care network member professional identity. To be effective in their role, the child psychotherapists needed to integrate the elements of their professional identity. The findings have implications for understanding the contribution of child psychotherapists in this field, particularly how to integrate into multi-disciplinary and multi-agency settings, whilst retaining their distinctive professional identity.

Generalization of Deep Reinforcement Learning for Jammer-Resilient Frequency and Power Allocation

We tackle the problem of joint frequency and power allocation while emphasizing the generalization capability of a deep reinforcement learning model. Most of the existing methods solve reinforcement learning-based wireless problems for a specific pre-determined wireless network scenario. The performance of a trained agent tends to be very specific to the network and deteriorates when used in a different network operating scenario (e.g., different in size, neighborhood, and mobility, among others). We demonstrate our approach to enhance training to enable a higher generalization capability during inference of the deployed model in a distributed multi-agent setting in a hostile jamming environment. With all these, we show the improved training and inference performance of the proposed methods when tested on previously unseen simulated wireless networks of different sizes and architectures. More importantly, to prove practical impact, the end-to-end solution was implemented on the embedded software-defined radio and validated using over-the-air evaluation.

Learning and Planning under Uncertainty for Conservation Decisions

My research focuses on new techniques in machine learning and game theory to optimally allocate our scarce resources in multi-agent settings to maximize environmental sustainability. Drawing scientific questions from my close partnership with conservation organizations, I have advanced new lines of research in learning and planning under uncertainty, inspired by the low-data, noisy, and dynamic settings faced by rangers on the frontlines of protected areas.

Physics-model-free heat-electricity energy management of multiple microgrids based on surrogate model-enabled multi-agent deep reinforcement learning

Reliability and cost-effectiveness in the operation of the multiple microgrid (MMG) system depend on the skillful management of its energy resources. Traditional energy management approaches are physics-model-based, which rely on the precise system parameters (e.g., line parameters) of the electricity and heat network. It is difficult to establish the precise system parameters in practice because they depend on a variety of factors. In this context, this paper proposes a physics-model-free control framework for the energy management of MMGs with heat-electricity energy, consisting of proposed surrogate model and multi-agent deep reinforcement learning (MADRL) approach. An important step is to use historical data to train a surrogate model in supervised manner that can imitate the realistic power and thermal flow calculations. Meanwhile, the energy management problem is reformulated as a Markov game. It is solved by the proposed MADRL-based approach by modeling each MG controller as an agent with a specific objective. The historical trajectories representation, parameter space technology, and deep dense architecture in reinforcement learning are introduced in MADRL to overcome the negative impact brought by the time series data from the input state on the decision-making process and construct an efficient exploration mechanism to overcome inefficient optimization of the MMG system in a multi-agent setting. During training period of MADRL, the trained surrogate models integrate into the environment of the MADRL, which can develop optimal energy management strategy based on the continuous interaction with the surrogate models. The proposed surrogate model enabled MADRL approach can reduce the reliance on precise physical systems and prevent having an impact on the real system while being trained involving trial and error process. Simulation results demonstrate the effectiveness of the proposed control framework.

Swarm Intelligence in Cooperative Environments: -Step Dynamic Tree Search Algorithm Overview

Reinforcement learning tree-based planning methods have been gaining popularity in the last few years due to their success in single-agent domains, where a perfect simulator model is available: for example, Go and chess strategic board games. This paper pretends to extend tree search algorithms to the multiagent setting in a decentralized structure, dealing with scalability issues and exponential growth of computational resources. The dynamic tree search combines forward planning and direct temporal-difference updates, outperforming markedly conventional tabular algorithms such as learning and state-action-reward-state-action (SARSA). Future state transitions and rewards are predicted with a model built and learned from real interactions between agents and the environment. This paper analyzes the developed algorithm in the hunter–pursuit cooperative game against stochastic and intelligent evaders. The dynamic tree search aims to adapt single-agent tree search learning methods to the multiagent boundaries and is demonstrated to be a remarkable advance as compared to conventional temporal-difference techniques.

Optimizing Mixed Autonomy Traffic Flow with Decentralized Autonomous Vehicles and Multi-Agent Reinforcement Learning

We study the ability of autonomous vehicles to improve the throughput of a bottleneck using a fully decentralized control scheme in a mixed autonomy setting. We consider the problem of improving the throughput of a scaled model of the San Francisco–Oakland Bay Bridge: a two-stage bottleneck where four lanes reduce to two and then reduce to one. Although there is extensive work examining variants of bottleneck control in a centralized setting, there is less study of the challenging multi-agent setting where the large number of interacting AVs leads to significant optimization difficulties for reinforcement learning methods. We apply multi-agent reinforcement algorithms to this problem and demonstrate that significant improvements in bottleneck throughput, from 20% at a 5% penetration rate to 33% at a 40% penetration rate, can be achieved. We compare our results to a hand-designed feedback controller and demonstrate that our results sharply outperform the feedback controller despite extensive tuning. Additionally, we demonstrate that the RL-based controllers adopt a robust strategy that works across penetration rates whereas the feedback controllers degrade immediately upon penetration rate variation. We investigate the feasibility of both action and observation decentralization and demonstrate that effective strategies are possible using purely local sensing. Finally, we open-source our code at https://github.com/eugenevinitsky/decentralized_bottlenecks .

A Delphi Study to Develop Items for a New Tool for Measuring Child Neglect for Use by Multi-Agency Practitioners in the UK

Social work and allied professions can struggle to accurately assess child neglect. Our research project is developing a new child neglect measurement tool for use by multi-agencies to address this issue. Phase two of this project employed a Delphi study to gather the views of a range of experts to help develop it. There were two important stages to inform the Delphi study: a systematic review of child neglect measures, and three online focus groups with a purposive sample of 16 participants with expertise in child neglect (academics, practitioners, and experts by experience). We then conducted a three-round modified online Delphi study with a purposive sample of 60 international panellists with expertise in child neglect. We followed the CREDES guidelines for the rigorous application of the Delphi technique. The panel generated salient items for the tool and scaled these for importance. The panel reached consensus for 18 items and 15 elements for the tool. The items included neglect type, chronicity, and severity. The elements included hyperlinks to research and the use of 10-point scales. The draft tool is short and may be useable by a range of practitioners in multi-agency settings. It is inclusive of social harms, such as poverty and social isolation. It will now be piloted.

A learning agent that acquires social norms from public sanctions in decentralized multi-agent settings

Society is characterized by the presence of a variety of social norms: collective patterns of sanctioning that can prevent miscoordination and free-riding. Inspired by this, we aim to construct learning dynamics where potentially beneficial social norms can emerge. Since social norms are underpinned by sanctioning, we introduce a training regime where agents can access all sanctioning events but learning is otherwise decentralized. This setting is technologically interesting because sanctioning events may be the only available public signal in decentralized multi-agent systems where reward or policy-sharing is infeasible or undesirable. To achieve collective action in this setting, we construct an agent architecture containing a classifier module that categorizes observed behaviors as approved or disapproved, and a motivation to punish in accord with the group. We show that social norms emerge in multi-agent systems containing this agent and investigate the conditions under which this helps them achieve socially beneficial outcomes.

Scaling multi-agent reinforcement learning to full 11 versus 11 simulated robotic football

Robotic football has long been seen as a grand challenge in artificial intelligence. Despite recent success of learned policies over heuristics and handcrafted rules in general, current teams in the simulated RoboCup football leagues, where autonomous agents compete against each other, still rely on handcrafted strategies with only a few using reinforcement learning directly. This limits a learning agent’s ability to find stronger high-level strategies for the full game. In this paper, we show that it is possible for agents to learn competent football strategies on a full 22 player setting using limited computation resources (one GPU and one CPU), from tabula rasa through self-play. To do this, we build a 2D football simulator with faster simulation times than the RoboCup simulator. We propose various improvements to the standard single-agent PPO training algorithm which help it scale to our multi-agent setting. These improvements include (1) using a policy and critic network with an attention mechanism that scales linearly in the number of agents, (2) sharing networks between agents which allow for faster throughput using batching, and (3) using Polyak averaged opponents, league opponents and freezing the opponent team when necessary. We show through experimental results that stable training in the full 22 player setting is possible. Agents trained in the 22 player setting learn to defeat a variety of handcrafted strategies, and also achieve a higher win rate compared to agents trained in the 4 player setting and evaluated in the full game.

Multi-Agent Deep Reinforcement Learning for Enhancement of Distributed Resource Allocation in Vehicular Network

To solve a decentralized radio resource management problem in a 5G vehicular network, we propose a novel resource allocation algorithm based on a multiagent deep reinforcement learning (MARL). We let each vehicle act as an individual agent that can select a unique combination of transport block (TB) and transmission power to broadcast periodic packets. Agent explores the environment and collects observations that later will be used to find the best combination of TB and transmission power. We apply an actor-critic reinforcement learning technique to choose optimal TB for each agent. To eliminate a nonstationarity in a multiagent setting, we utilize a centralized training that allows all agents to share their observations over critic networks. The information shared through critic network can assist each agent to learn the policies of other agents. In a decentralized execution, each agent may only use its actor network and local observation to find the most appropriate TB in the given level of transmission power. While training, the actions taken by actor are evaluated by corresponding critic that maps Q-value for all feasible actions in the given state. Our method results in 18% higher packet reception ratio than a spectrum allocation scheme based on a double DQN. The proposed method achieves 33% higher reward than the previous state-of-the-art that is also based on MARL.

Online learning of energy consumption for navigation of electric vehicles

Energy efficient navigation constitutes an important challenge in electric vehicles, due to their limited battery capacity. We employ a Bayesian approach to model the energy consumption at road segments for efficient navigation. In order to learn the model parameters, we develop an online learning framework and investigate several exploration strategies such as Thompson Sampling and Upper Confidence Bound. We then extend our online learning framework to the multi-agent setting, where multiple vehicles adaptively navigate and learn the parameters of the energy model. We analyze Thompson Sampling and establish rigorous regret bounds on its performance in the single-agent and multi-agent settings, through an analysis of the algorithm under batched feedback. Finally, we demonstrate the performance of our methods via experiments on several real-world city road networks.

ZO-JADE: Zeroth-Order Curvature-Aware Distributed Multi-Agent Convex Optimization

In this work we address the problem of convex optimization in a multi-agent setting where the objective is to minimize the mean of local cost functions whose derivatives are not available (e.g. black-box models). Moreover agents can only communicate with local neighbors according to a connected network topology. Zeroth-order (ZO) optimization has recently gained increasing attention in federated learning and multi-agent scenarios exploiting finite-difference approximations of the gradient using from $2$ (directional gradient) to $2d$ (central difference full gradient) evaluations of the cost functions, where $d$ is the dimension of the problem. The contribution of this work is to extend ZO distributed optimization by estimating the curvature of the local cost functions via finite-difference approximations. In particular, we propose a novel algorithm named ZO-JADE, that by adding just one extra point, i.e. $2d+1$ in total, allows to simultaneously estimate the gradient and the diagonal of the local Hessian, which are then combined via average tracking consensus to obtain an approximated Jacobi descent. Guarantees of semi-global exponential stability are established via separation of time-scales. Extensive numerical experiments on real-world data confirm the efficiency and superiority of our algorithm with respect to several other distributed zeroth-order methods available in the literature based on only gradient estimates.

An epistemic logic for formalizing group dynamics of agents

Abstract In the multi-agent setting, it is relevant to model group dynamics of agents, and logic has proved a good tool to do so. We propose an epistemic logic, L-DINF-E, that allows one to formalize what are the beliefs formed by a group of agents, where several groups exist and agents can pass from a group to another one. We introduce a new modality which allows an agent to reason about the beliefs of other agents. This allows us to model aspects of the “Theory of Mind”, understood as the set of social-cognitive skills involving the ability to attribute and reason about mental states, desires, beliefs, and knowledge of agents. In this paper, we present the logic L-DINF-E and illustrate how it can be used to solve “false-belief tasks”, i.e., tests in which an agent should understand that some other agent may develop, under some circumstances, false beliefs.

Deep Reinforcement Learning-Based Operation of Transmission Battery Storage with Dynamic Thermal Line Rating

It is well known that dynamic thermal line rating has the potential to use power transmission infrastructure more effectively by allowing higher currents when lines are cooler; however, it is not commonly implemented. Some of the barriers to implementation can be mitigated using modern battery energy storage systems. This paper proposes a combination of dynamic thermal line rating and battery use through the application of deep reinforcement learning. In particular, several algorithms based on deep deterministic policy gradient and soft actor critic are examined, in both single- and multi-agent settings. The selected algorithms are used to control battery energy storage systems in a 6-bus test grid. The effects of load and transmissible power forecasting on the convergence of those algorithms are also examined. The soft actor critic algorithm performs best, followed by deep deterministic policy gradient, and their multi-agent versions in the same order. One-step forecasting of the load and ampacity does not provide any significant benefit for predicting battery action.

Hybrid Multiagent Reinforcement Learning for Electric Vehicle Resilience Control Towards a Low-Carbon Transition

In responseto low-carbon requirements, a large amount of renewable energy resources (RESs) have been deployed in power systems; nevertheless, the intermittency of RESs raises the system vulnerability and even causes severe damage under extreme events. Electric vehicles (EVs), owing to their mobility and flexibility characteristics, can provide various ancillary services meanwhile enhancing system resilience. The distributed control of EVs under such scenarios in power-transportation network becomes a complex decision-making problem with enormous dynamics and uncertainties. To this end, a multiagent reinforcement learning method is proposed to compute both discrete and continuous actions simultaneously that aligns with the nature of EV routing and scheduling problems. Furthermore, the proposed method can enhance the learning stability and scalability with privacy perseverance in the multiagent setting. Simulation results based on IEEE 6- and 33-bus power networks integrated with transportation systems validate its effectiveness in providing system resilience and carbon intensity service.

Deep multiagent reinforcement learning: challenges and directions

This paper surveys the field of deep multiagent reinforcement learning (RL). The combination of deep neural networks with RL has gained increased traction in recent years and is slowly shifting the focus from single-agent to multiagent environments. Dealing with multiple agents is inherently more complex as (a) the future rewards depend on multiple players’ joint actions and (b) the computational complexity increases. We present the most common multiagent problem representations and their main challenges, and identify five research areas that address one or more of these challenges: centralised training and decentralised execution, opponent modelling, communication, efficient coordination, and reward shaping. We find that many computational studies rely on unrealistic assumptions or are not generalisable to other settings; they struggle to overcome the curse of dimensionality or nonstationarity. Approaches from psychology and sociology capture promising relevant behaviours, such as communication and coordination, to help agents achieve better performance in multiagent settings. We suggest that, for multiagent RL to be successful, future research should address these challenges with an interdisciplinary approach to open up new possibilities in multiagent RL.

Water distribution in community irrigation using a multi-agent system

ABSTRACT Insufficient water for irrigation is a common problem in the Canterbury Region of New Zealand. Farmers have the option of applying for resource consent or joining a community irrigation scheme to take water. Water supply becomes more problematic during drought seasons as farmers must adhere to water restrictions imposed by the water authority. To deal with this problem, we developed an agent-based irrigation management system that can be used by farmers to calculate the ideal crop water needs on individual farms, which is particularly useful during periods of water scarcity. During water scarcity, most farms will have shortages of water. However, it is possible that there are farmers who will have excess water that could be distributed to those who need it. By doing this, farmers with excess water can make more profit and those who do not have enough water can purchase water to reduce their losses. In this work, we explore how auction-based negotiation in a multi-agent setting can be used to maximise water sharing within a community during periods of water scarcity. We evaluate various auction mechanisms that can be used to distribute excess water. In addition, we investigate the effect of various different agents' behaviours on water distribution and community profit.

Investigation of independent reinforcement learning algorithms in multi-agent environments.

Independent reinforcement learning algorithms have no theoretical guarantees for finding the best policy in multi-agent settings. However, in practice, prior works have reported good performance with independent algorithms in some domains and bad performance in others. Moreover, a comprehensive study of the strengths and weaknesses of independent algorithms is lacking in the literature. In this paper, we carry out an empirical comparison of the performance of independent algorithms on seven PettingZoo environments that span the three main categories of multi-agent environments, i.e., cooperative, competitive, and mixed. For the cooperative setting, we show that independent algorithms can perform on par with multi-agent algorithms in fully-observable environments, while adding recurrence improves the learning of independent algorithms in partially-observable environments. In the competitive setting, independent algorithms can perform on par or better than multi-agent algorithms, even in more challenging environments. We also show that agents trained via independent algorithms learn to perform well individually, but fail to learn to cooperate with allies and compete with enemies in mixed environments.

From motor control to team play in simulated humanoid football.

Learning to combine control at the level of joint torques with longer-term goal-directed behavior is a long-standing challenge for physically embodied artificial agents. Intelligent behavior in the physical world unfolds across multiple spatial and temporal scales: Although movements are ultimately executed at the level of instantaneous muscle tensions or joint torques, they must be selected to serve goals that are defined on much longer time scales and that often involve complex interactions with the environment and other agents. Recent research has demonstrated the potential of learning-based approaches applied to the respective problems of complex movement, long-term planning, and multiagent coordination. However, their integration traditionally required the design and optimization of independent subsystems and remains challenging. In this work, we tackled the integration of motor control and long-horizon decision-making in the context of simulated humanoid football, which requires agile motor control and multiagent coordination. We optimized teams of agents to play simulated football via reinforcement learning, constraining the solution space to that of plausible movements learned using human motion capture data. They were trained to maximize several environment rewards and to imitate pretrained football-specific skills if doing so led to improved performance. The result is a team of coordinated humanoid football players that exhibit complex behavior at different scales, quantified by a range of analysis and statistics, including those used in real-world sport analytics. Our work constitutes a complete demonstration of learned integrated decision-making at multiple scales in a multiagent setting.