Multi-agent Game Research Articles

The evolution analysis of low-carbon power transition strategies and carbon emission decoupling based on carbon neutrality target

It is necessary for China to accelerate the low-carbon power transition to realize the carbon neutrality target by 2060. Based on the goal of carbon neutrality, Deep Reinforcement Learning with the Double Q-learning method is used to solve a multi-agent game in 30 provinces of China, in an attempt to explore the evolutionary process of regional low-carbon power transition strategies, carbon emissions, and carbon emission decoupling with economic growth. Conclusions are obtained as follows. First, regional low-carbon power transition strategies are not consistent. Though China is bound to achieve the carbon peak and carbon neutrality, the carbon peak and carbon neutrality process will vary widely across regions and over time. Second, the cost of power decarbonization is higher in resource-based regions than that in core regions. The regional coordination policy can reduce the total carbon emissions in resource-based regions, but it will not alleviate their carbon emission decoupling costs. Third, the stable strategy of low-carbon power transition needs faster economic development and more abundant zero-carbon resources. Therefore, the carbon neutrality policy is suggested to be more spatially differentiated, more equitable, and more robust.

A fast strategy-solving method for adversarial team games utilizing warm starting

Adversarial team games (ATGs) have garnered significant attention in recent years, leading to the emergence of various solutions such as linear programming algorithms, multi-agent reinforcement learning, and game tree transformation. The ATGs involve large-scale game trees, resulting in higher time costs for computation. In this paper, we focus on expediting the solution of team-maxmin equilibrium with correlation (TMECor), which can be considered the equilibrium that maximizes the team’s payoff. To address this, we propose a transformation with seed strategies (TSS). TSS leverages reinforcement learning to calculate player strategies. We initialize the strategies of all players, referred to as seed strategies, and incorporate them into the multi-agent game tree during the transformation process. These seed strategies serve as the starting strategies for counterfactual regret minimization (CFR). CFR initializes the strategies and cumulative regret of all players based on the seed strategy. By warm starting the whole process, our method accelerates the solving of TMECor. We conducted nine experiments using Kuhn poker and Leduc Hold’em poker. The results demonstrated that TSS improved the solving speed of TMECor.

A Review of Multi-Agent Game in Intellectual Property Pledge Financing

To promote the theoretical research and business development of intellectual property pledge financing, this paper summarizes the research status of multi-agent game in intellectual property pledge financing. At present, scholars’ research on multi-agent game in intellectual property pledge financing mainly focuses on the game between borrowers and lenders, the game between lenders and third-party intermediaries (or platforms), and the game between borrowers, lenders and third-party intermediaries (or platforms). Overall, scholars have conducted extensive research on the game between borrowers and lenders, while research on the game between borrowers, lenders, and third-party intermediaries (or platforms) is relatively weak. Especially when the government implements financial subsidies and interest subsidies for borrowers, financial subsidies for third-party intermediaries (or platforms), and risk compensation for lenders, there is a lack of research on the intellectual property pledge financing game with the government as an independent game subject. Future research can focus on the game between borrowers, lenders, and the government under government incentives, as well as the game between borrowers, lenders, third-party intermediaries (or platforms) and the government under government incentives.

Bi-level game theoretic approach for robust design: A case study of path-generating four-bar

This study addresses the bi-level multi-objective optimization problems (MOP) that raise in robust design and optimization of engineering systems through establishing a state-of-the-art game theoretic scenario. A novel leader-follower decentralized decision-making scenario is proposed, leveraging the synergy of game theory, Robust Design Optimization (RDO), Monte Carlo Simulation (MCS), and Artificial Intelligence (AI). The proposed algorithm can be employed for optimum robust Pareto design of a wide range of dynamical systems. In order to achieve a robust design, both the mean and variance of each objective function are considered as players in a multi-agent game setting. In this approach, both Stackelberg and cooperative games are utilized to model the behaviors of the players. Genetic Programming (GP) meta-models are employed to capture the Stackelberg protocol between two levels specifically for constructing the follower's rational reaction set (RRS). Additionally, the Nash bargaining function is -utilize to model the cooperative behaviors among players in each level. The proposed approach is applied and demonstrated through a case study involving multi-objective robust design of planar four-bar linkages. In this manner, four objective functions are assigned to four players within the system. Each player is responsible for optimizing a specific objective criterion, namely the mean of tracking error (TE), variance of tracking error, mean of transmission angle and variance of transmission angle (TA) of the linkage. As a result, the four-objective optimization problem of mechanism is transformed into a single-objective robust synthesis problem. The comparisons of the results show a significant enhancement in the robust behavior of the linkage, while ensuring that deterministic criteria such as quality of motion and precision are preserved.

Low-carbon economic operation strategy for multi-agent integrated energy system considering uncertainty of renewable energy power generation

The uncertainty of renewable energy output threatens the operation safety of multi-agent integrated energy system (MAIES), which makes it difficult to balance the low-carbon economic operation demands of various stakeholders. However, the existing research solely focuses on the operational strategy of multi-agent game involving integrated energy suppliers and users in deterministic scenarios, overlooking the complementary supporting role and game interaction of shared energy storage and wind farm as independent entities of interest under the instability of renewable energy power generation. Hence, this paper first establishes the optimal operation models for integrated energy system operator (IESO), user aggregator (UA), shared energy storage operator (SESO), and wind farm operator (WFO) considering the stepped carbon trading. Second, in the face of the actual situation of uncertainty of photovoltaic and wind power output, fuzzy chance-constrained programming is adopted for processing. Then, a bi-layer game equilibrium model with IESO as a leader and UA, SESO, and WFO as followers is proposed, and the existence and uniqueness of Stackelberg equilibrium solution are proved. Finally, simulation calculation is carried out based on the YALMIP toolbox in the Matlab R2023a software, and the improved particle swarm optimization algorithm and CPLEX solver are used to solve the model. The results demonstrate that the participation of SESO and WFO as independent stakeholders in the game interaction can improve the economic and environmental benefits of MAIES. The iterative optimization of demand response subsidy prices can effectively motivate users to participate in demand response, improve the ability of MAIES to cope with the uncertain risks of renewable energy generation and load, and reduce the power grid dispatch pressure.

Co-evolutionary dynamics in optimal multi-agent game with environment feedback

At present, game-based analysis and control problems with environmental feedback significantly contribute to the study of collective cooperation, and pose great challenges for the analysis of multi-agent game systems in complex real world. We consider co-evolutionary dynamics based on an extended multi-agent game with environmental feedback, which is modeled by optional public goods games (OPGGs). Firstly, the co-evolutionary dynamics of OPGGs with environment feedback can be modeled as two parts: the dynamics of OPGGs and synergy coefficient, where the former can represent the group game behavior of three strategies, and the latter can be seen as the evolutionary behavior in changing environments. Subsequently, some possible fixed points will emerge with respect to three types of boundaries for these co-evolutionary dynamics, and the existence and stability for the corresponding boundary fixed points are determined and analyzed. Meanwhile, the results are further extended to interior equilibrium point. It is observed that by combining voluntary participation and environment feedback, cooperation behavior can be still feasible, and the evolving system can oscillatingly converge to the persistent cooperation under some certain threshold conditions. Finally, an example is illustrated to validate the proposed approach. To sum up, environmental feedback mechanism provides a novel perspective to investigate the collective cooperation in a changing world.

Research on the strategy for improving the utility of government social media information based on a multi-agent game model

Government social media (GSM) has become an important tool for government departments to open information, guide public opinion and interact with the government and the people. However, the operation and maintenance of some GSM are not standardised, and the content published is inconsistent with identity positioning, resulting in the realistic dilemma of low utility of GSM information. The purpose of this study is to explore the effective strategies to improve the effectiveness of GSM information. The research is from the perspective of information economics, this article uses evolutionary game theory to build a tripartite evolutionary game model comprising GSM operations departments, government regulators and users in order to explore the evolution process of tripartite game behaviours and the influence of subject behaviour selection on information utility. It subsequently conducts a solution and numerical simulation to demonstrate the influence of different factors on the game results. The experimental results show that there are four situations in which the utility of GSM information affects the evolution and stability strategy of the subject and that changes in different parameter values have significant effects on the results of the three-party game. The evolution trend of the subject behaviour can be changed by increasing the regulatory means of rewards and punishments and establishing an efficient operation mechanism for GSM, thus promoting system convergence to the ideal state. The results of this study can provide references and suggestions for government departments to effectively enhance the effectiveness of GSM information and promote the healthy development of GSM.

Embedding multi-agent reinforcement learning into behavior trees with unexpected interruptions

Behavior trees have attracted great interest in computer games and robotic applications. However, it lacks the learning ability for dynamic environments. Previous works combining behavior trees with reinforcement learning either need to construct an independent sub-scenario or train the learning method over the whole game, which is not suited for complex multi-agent games. In this paper, a framework is proposed, named as MARL-BT, that embeds multi-agent reinforcement learning methods into behavior trees. Following the running mechanism of behavior trees, we design the way of collecting samples and the training procedure. Further, we point out a special phenomenon in MARL-BT, i.e., the unexpected interruption, and present an action masking technique to remove its harmful effect on learning performance. Finally, we make extensive experiments on the 11 versus 11 full game in Google Research Football. The introduced MARL-BT framework could get an 11.507% improvement compared to pure BT for certain scenarios. The action masking technique could greatly improve the performance of the learning method, i.e., the final reward is improved around 100% times for a sub-task.

DVF:Multi-agent Q-learning with difference value factorization

In cooperative multi-agent games, agents are required to learn effective cooperative behaviors within complex action spaces. An effective approach is to utilize the Individual-Global-Max (IGM) principle to decompose the global reward signal into individual contributions from each agent. However, existing methods confront two significant challenges: ensuring the accuracy of the value function decomposition process and guaranteeing monotonic improvement in joint policies. These challenges become exacerbated in scenarios involving non-monotonic matrices. To address these challenges, we introduce a novel and flexible value factorization method called Difference Value Factorization (DVF). The key idea of our method is to transform the IGM principle into a new form DVF-IGM, which addresses non-monotonic constraints by ensuring the consistency of the IGM process between the joint difference value and the complex non-linear sum of independent difference values. A centralized evaluator is employed to estimate global Q-values, which not only enhances expressiveness but also constructs difference values for updating individual value functions. We demonstrate that DVF-IGM is an equivalent transformation of IGM and that DVF has the monotonic improvement property. Empirically, our method has been shown to maintain and recover the optimal policy in non-monotonic matrix games and achieve state-of-the-art performance in cooperative tasks within the StarCraft Multi-Agent Challenge (SMAC).

Cooperative multi-agent game based on reinforcement learning

Multi-agent reinforcement learning holds tremendous potential for revolutionizing intelligent systems across diverse domains. However, it is also concomitant with a set of formidable challenges, which include the effective allocation of credit values to each agent, real-time collaboration among heterogeneous agents, and an appropriate reward function to guide agent behavior. To handle these issues, we propose an innovative solution named the Graph Attention Counterfactual Multiagent Actor–Critic algorithm (GACMAC). This algorithm encompasses several key components: First, it employs a multi-agent actor–critic framework along with counterfactual baselines to assess the individual actions of each agent. Second, it integrates a graph attention network to enhance real-time collaboration among agents, enabling heterogeneous agents to effectively share information during handling tasks. Third, it incorporates prior human knowledge through a potential-based reward shaping method, thereby elevating the convergence speed and stability of the algorithm. We tested our algorithm on the StarCraft Multi-Agent Challenge (SMAC) platform, which is a recognized platform for testing multi-agent algorithms, and our algorithm achieved a win rate of over 95% on the platform, comparable to the current state-of-the-art multi-agent controllers.

Reducing Q-Value Estimation Bias via Mutual Estimation and Softmax Operation in MADRL

With the development of electronic game technology, the content of electronic games presents a larger number of units, richer unit attributes, more complex game mechanisms, and more diverse team strategies. Multi-agent deep reinforcement learning shines brightly in this type of team electronic game, achieving results that surpass professional human players. Reinforcement learning algorithms based on Q-value estimation often suffer from Q-value overestimation, which may seriously affect the performance of AI in multi-agent scenarios. We propose a multi-agent mutual evaluation method and a multi-agent softmax method to reduce the estimation bias of Q values in multi-agent scenarios, and have tested them in both the particle multi-agent environment and the multi-agent tank environment we constructed. The multi-agent tank environment we have built has achieved a good balance between experimental verification efficiency and multi-agent game task simulation. It can be easily extended for different multi-agent cooperation or competition tasks. We hope that it can be promoted in the research of multi-agent deep reinforcement learning.

Multi-agent game operation of regional integrated energy system based on carbon emission flow

In the process of promoting energy green transformation, the optimization of regional integrated energy system faces many challenges such as cooperative management, energy saving and emission reduction, as well as uncertainty of new energy output. Therefore, this paper proposes a multi-agent game operation method of regional integrated energy system based on carbon emission flow. First, this paper establishes a carbon emission flow calculation model for each subject, and proposes a comprehensive tariff model based on the carbon emission flow, which discounts the carbon emissions from the power supply side to the power consumption side. Secondly, considering the interests of each subject, this paper establishes the decision-making model of each subject. And the new energy uncertainty, the cost of energy preference of prosumers, and the thermal inertia of buildings are considered in the decision model. Finally, the model is solved using differential evolution algorithm and solver. The case study verifies that the comprehensive electricity pricing model based on carbon emission flow developed in this paper can play a role in balancing economy and low carbon.

Research on Industrial Economic Development and Electricity Price Adjustment Strategy Based on Multi-Agent Game

With the deepening of the electricity market reform and the accelerating transformation of the new power system, the number of new electricity customers in the market with local development characteristics has increased. The local industrial and economic development have been affected by the ability to guide electricity prices. In this paper, using the multi-agent game based on exploring the impact of electricity price on the industrial economy, the specific optimization scheme of electricity price adjustment is proposed with the aim of improving the overall efficiency of the power grid and reducing the cost of electricity for industrial users. After the game, the overall efficiency of the electricity supplier will be increased by about 3.06 percent, and the cost of electricity used by various special industries will be reduced by about 6.86 percent to 4.48 percent. The win-win situation of the electricity suppliers’ income and the featured industries’ development can be achieved by the optimization results. As well as the cost of electricity consumption in featured industries can be reduced, and the comprehensive benefits of electricity suppliers can be improved by using the electricity price adjustment scheme after game interaction, which promotes the rational use of energy and meets the current social requirements for the green and sustainable development of the electricity industry.

Distributed Nash equilibrium searching for multi-agent games under false data injection attacks

In this paper, for the multi-agent system, the Nash equilibrium (NE) search problem for non-cooperative game with state constraints under false data injection (FDI) attack is studied. The FDI attack can directly destroy the connectivity between agents, resulting in the performance degradation or even failure for most NE search algorithms. We propose a new distributed NE search method to improve the performance of NE algorithms under FDI attacks. The NE algorithm is designed to find the solution of NE under network attack and make the system recover after the attack. However, most of the existing results only studied the NE algorithm of the non-attacking system, and the existing results rarely take into account whether the system state exceeds the safety limit, which can cause the system to crash. This paper solves the state constraints problem of the multi-agent game under the FDI attack. Finally, the reliability of the algorithm is verified by a numerical result.

Peer Incentive Reinforcement Learning for Cooperative Multiagent Games

Social learning, especially social incentives, is extremely important for humans to achieve a high level of coordination. Inspired by this, we introduce this concept into cooperative multi-agent reinforcement learning (MARL), to implicitly address the credit assignment problem and promote the inter-agent direct interactions for cooperations among agents in cooperative multi-agent games. In this paper, we propose a novel Intrinsic Reward method with Peer Incentives (IRPI) based on actor-critic policy gradient. This method can enable agents to incentivize each other for their cooperations through using causal influence among them. Specifically, a novel intrinsic reward mechanism is innovatively designed to empower each agent the ability to give positive or negative rewards to other peer agents' actions through considering the causal influence of the other agents on it. The mechanism is realized by a feed-forward neural network through utilizing causal influence between the agents. The causal influence of one agent on another is inferred via counterfactual reasoning using the joint action-value function in MARL. The quality of the influence is assessed via counterfactual reasoning using the individual value function in MARL. Simulations are carried out on two popular multi-agent game testbeds: Starcraft II Micromanagement and Multi-Agent Particle Environments. Simulational results demonstrate that the proposed IRPI can enhance cooperations among the agents to achieve better performance compared with a number of state-of-the-art MARL methods in a variety of cooperative multi-agent games.

Generating intelligent agent behaviors in multi-agent game AI using deep reinforcement learning algorithm

<span>The utilization of games in training the reinforcement learning (RL) agent is to describe the complex and high-dimensional real-world data. By utilizing games, RL researchers will be able to evade high experimental costs in training an agent to do intelligence tasks. The objective of this research is to generate intelligent agent behaviors in multi-agent game artificial intelligence (AI) using deep reinforcement learning (DRL) algorithm. A basic RL algorithm called deep Q network is chosen to be implemented. The agent is trained by the environment's raw pixel images and the action list information. The experiments conducted by using this algorithm show the agent’s decision-making ability in choosing a favorable action. In the default setting for the algorithm, the training is set into 1 epoch and 0.0025 learning rate. The number of training iterations is set to one as the training function will be repeatedly called for every 4-timestep. However, the author also experimented with two different scenarios in training the agent and compared the results. The experimental findings demonstrate that our agents learn correctly and successfully while actively participating in the game in real time. Additionally, our agent can quickly adjust against a different enemy on a varied map because of the observed knowledge from prior training.</span>

Overview on reinforcement learning of multi-agent game

Game intelligence is an emerging hot topic in the field of artificial intelligence in recent years, and multi-agent learning is a frontier topic in the field of the intelligent game, which has a huge development prospect in all fields. This paper introduces the origin of reinforcement learning (RL) from the law of effect in animal experimental psychology and the optimization theory of optimal control. Then, the author describes the systematic composition of multi-agent reinforcement learning (MARL), and summarizes the classification of its research methods. The existing problems of MARL are discussed from three aspects: non-stationarity of the environment, partial observability, and the dimensional explosion problem. Finally, an outlook on the future is given based on the current development status of MARL and the important and difficult issues in the research field.

An Augmented Game Approach for Design and Analysis of Distributed Learning Dynamics in Multiagent Games.

In this article, an augmented game approach is proposed for the formulation and analysis of distributed learning dynamics in multiagent games. Through the design of the augmented game, the coupling structure of utility functions among all the players can be reformulated into an arbitrary undirected connected network while the Nash equilibria are preserved. In this case, any full-information game learning dynamics can be recast into a distributed form, and its convergence can be determined from the structure of the augmented game. We apply the proposed approach to generate both deterministic and stochastic distributed gradient play and obtain several negative convergent results about the distributed gradient play: 1) a Nash equilibrium is convergent under the classic gradient play, yet its corresponding augmented Nash equilibrium may be not convergent under the distributed gradient play and, on the other side, 2) a Nash equilibrium is not convergent under the classic gradient play, yet its corresponding augmented Nash equilibrium may be convergent under the distributed gradient play. In particular, we show that the variational stability structure (including monotonicity as a special case) of a game is not guaranteed to be preserved in its augmented game. These results provide a systematic methodology about how to formulate and then analyze the feasibility of distributed game learning dynamics.

Behavior Learning System for Robot Soccer Using Neural Network

With technological developments, the prospect of a human-robot symbiotic society has emerged. A soccer game has characteristics similar to those expected in such a society. Soccer is a multiagent game in which the strategy employed depends on each agent’s position and actions. This paper discusses the results of the development of a learning system that uses a self-organizing map to select behaviors depending on the scenario (two-dimensional absolute coordinates of the agent, other agents, and the ball). The system can reproduce the action-selection algorithms of all the players on a certain team, and the robot can instantly select the next cooperative action from information obtained during the game. Thus, common-sense rules can be shared to learn an action-selection algorithm for a set of both human and robot agents.

Game-Theoretic Decision-Making and Payoff Design for UAV Collision Avoidance in a Three-Dimensional Airspace

Safety and efficiency are primary goals of air traffic management. With the integration of unmanned aerial vehicles (UAVs) into the airspace, UAV traffic management (UTM) has attracted significant interest in the research community to maintain the capacity of three-dimensional (3D) airspace, provide information, and avoid collisions. We propose a new decision-making architecture for UAVs to avoid collision by formulating the problem into a multi-agent game in a 3D airspace. In the proposed game-theoretic approach, the Ego UAV plays a repeated two-player normal-form game, and the payoff functions are designed to capture both the safety and efficiency of feasible actions. An optimal decision in the form of Nash equilibrium (NE) is obtained. Simulation studies are conducted to demonstrate the performance of the proposed game-theoretic collision avoidance approach in several representative multi-UAV scenarios.