Decentralized Counterfactual Value with Threat Detection for Multi-Agent Reinforcement Learning in mixed cooperative and competitive environments

Abstract

This paper proposes a fully decentralized approach to address the challenge of general mixed cooperation and competition within the domain of Multi-Agent Reinforcement Learning (MARL). Conventional MARL approaches do not achieve full decentralization as they necessitate either the communication of implicit information or the retention of a centralized critic, rendering them impractical in mixed cooperative and competitive environments. To address these challenges, this paper proposes a Decentralized Counterfactual Value (DCV) to model the behaviors of other agents and mitigate the non-stationary problem, accompanied by a Threat Detection (TD) mechanism to discern latent competitive or cooperative relationships. In addition, DCVTD is incorporated into both value-based and policy-based RL paradigms with theoretical convergence guarantee. Finally, empirical validation across four representative environments demonstrates the superior performance of DCVTD in terms of collective returns, computational efficiency, and agent scalability over other fully decentralized approaches, centralized training with decentralized execution approaches, and alternative approaches involving agent modeling or reward shaping in comprehensive experiments.