Deep reinforcement learning for multi-objective game strategy selection

Abstract

Multi-objective game (MOG) is a fundamental model for the decision-making problems in which each player must consider multi-dimensional payoffs that reflect different objectives. Typically, solving MOG involves refining the set of equilibrium strategies, which is also known as MOG strategy selection (MOGS). However, existing MOG algorithms only allow one metric for MOGS, which limits the application in real-world scenarios where the players may have different preferences over multiple metrics. In this paper, we first develop a preference-based MOGS framework to encompass multiple metrics with different preferences in MOGS. Based on the framework, we introduce the concept of comprehensive evaluation value (CEV) to evaluate the quality of a strategy set given the preference of each metric. Using CEV as a reward signal, we formulate the problem of finding the optimal strategy set as a Markov decision process, and use deep reinforcement learning to train a policy for MOG strategy selection given the metrics and the corresponding preferences. Specifically, we combine a rational strategy filtering procedure with a Transformer-based encoder–decoder policy network to refine the strategies given the preferences, and then we use a revised REINFORCE algorithm to train the policy network. Besides, we introduce variable beam search decoding to improve the quality of a rollout by keeping track of the most promising strategy sets and choosing the best one. We benchmark our algorithm on the MOG instances generated by GAMUT, and extensive experiments demonstrate that our algorithm can generate the strategy set significantly better than the state-of-the-art baselines with lower computational overhead given different preferences. Furthermore, we compare our approach on real-world problems, showing the great advantages in both performance and runtime.