Multi-agent Game Research Articles

Multi-objective scheduling in dynamic of household paper workshop considering energy consumption in production process

Aim: The uncertainty and complexity of the production process of household paper are growing sharply in modern factories. Due to the influence of rising energy costs and environmental policies, the demand for reducing production costs and energy consumption is also increasing. Therefore, it is studied that the dynamic shop scheduling problem of household paper production considering simultaneously the cost with energy consumption. Methods: A mathematical model of the multi-objective and multi-constraint household paper scheduling problem is established first. The multi-objective scheduling process is transformed into a multi-agent Markov game process by assigning each objective to each agent. Upon which, a multi-agent game model is constructed for the household paper scheduling problem based on deep reinforcement learning; it is a proposed D3QN algorithm and Nash equilibrium strategy, and the state characteristics and action selection space are proposed according to the characteristics of the household paper production. The model performance has been verified by the actual production data. Results: Results show that the proposed method not only achieves better performance than traditional scheduling methods but also persists in its advantages even when the configuration of the manufacturing system changes. Conclusion: Multi-agent deep reinforcement learning in the Markov game has a good prospect in solving multi-objective dynamic scheduling problems for household paper production.

A logic Petri net model for dynamic multi-agent game decision-making

This study proposes a logical Petri net model to leverage the modeling advantages of Petri nets in handling batch processing and uncertainty in value passing and to integrate relevant game elements from multi-agent game processes for modeling multi-agent decision problems and resolving optimization issues in dynamic multi-agent game decision-making. Firstly, the attributes of each token are defined as rational agents, and utility function values and state probability transition functions are assigned to them. Secondly, decision transitions are introduced, and the triggering of the optimal decision transition is determined based on a comparison of token utility function values, along with an associated algorithm. Finally, a dynamic game emergency business decision-making process for sudden events is modeled and analyzed using the logic game decision Petri net. Based on reachable markings, reachable graphs are constructed to analyze the dynamic game process. Algorithms are described for the generation of reachable graphs, and the paper explores how the logic game decision model for sudden events can address dynamic game decision problems, generate optimal emergency plans, and analyze resource conflicts during emergency processes. The effectiveness and superiority of the model in analyzing the emergency business decision-making process for sudden events are validated.

Cognition-Driven Multiagent Policy Learning Framework for Promoting Cooperation

Many attempts have been made to promote cooperation for multi-agent systems. However, several issues that draw less attentions but may dramatically degrade the cooperation performance still exist, such as redundant information interactions among neighbors, and difficulties in understanding complex and dynamic environments from high-level cognition. To address these limitations, a cognition-driven multi-agent policy (CDMAP) learning framework is proposed in this paper. It includes a cognition difference network (CDN), a coupling cognition network (CCN), and a policy optimization network (PON). CDN is designed based on a variational auto-encoder, where a concept of cognition difference is defined to prune redundant interactions among agents for more efficient communication. Based on the pruned topology, CCN captures the hidden representations of the surrounding environment. The novelty lies in that several coupling graph attention layers are incorporated, each graph with different but coupling adjacent matrices, yielding a comprehensive state understanding from multiple representation spaces. Based on the captured hidden states, PON generates the final policies, where QMIX is adopted as a value factorization method to alleviate the credit-assignment problem. At last, CDMAP is evaluated through two representative multi-agent games including Google Research Football and StarCraft II. The results demonstrate its superior effectiveness compared with existing methods. Animated demonstrations and performance video of the learned strategy are provided in <uri>https://sites.google.com/view/cdmap</uri>.

A Multiagent Game Theoretic Simulation of Public Policy Coordination through Collaboration

Background: Policy coordination is necessary to address many of the sustainability challenges we face today. The formal representations of policy coordination focus on modeling conflict management but neglect its collaborative nature. This limits efforts to build more realistic models of policy coordination. The objective of this paper is to simulate collaboration and noncollaboration between agents in the context of policy coordination in order to determine the effect of different approaches to policy coordination. Methods: For this purpose, a multiagent simulation of collaboration based on evolutionary game theory is used. Results: The results suggest that policy coordination through collaboration produces the most desirable outcomes and that reducing the cost of communication between agents is necessary to increase the probability of collaboration. Conclusions: The cost of information (both its transmission and transformation) is critical to increase the probability of collaboration in policy coordination. This paper advances the understanding of how to model the collaborative nature of policy coordination by contributing to the methodological standardization of the analysis and implementation of public policy coordination.

Planning distribution network using the multi-agent game and distribution system operators

When planning the distribution network, the income of each market entity is calculated by a fixed price. How to take the price of power into account while developing the planning strategy for each organization in the actual power market is an urgent issue that needs to be addressed imminently. To address this problem, considering the continuous change in the market price due to the change in the supply–demand relationship in the actual power market, this article proposes a distribution network planning method which considers the distribution system operator (DSO) and multi-agent game. First, the planning decision model of distribution network companies and power users with different interest subjects is constructed with grid planning and DG operation as decision variables. Second, DSO is introduced to the game framework. Based on the distribution locational marginal pricing (DLMP), a price accounting model is being developed. Then, the transfer relationships and game behavior among the distribution company, power users, and DSO are analyzed. Finally, an iterative search algorithm is used to solve a multi-agent game planning model of a distribution network that takes into account price signals in the power market. Examples based on IEEE 33-bus systems validate the suggested method’s validity and efficacy.

Modeling and Prediction of Sustainable Urban Mobility Using Game Theory Multiagent and the Golden Template Algorithm

The current development of multimodal transport networks focuses on the realization of intelligent transport systems (ITS) to manage the prediction of traffic congestion and urban mobility of vehicles and passengers so that alternative routes can be recommended for transport, especially the use of public passenger transport, to achieve sustainable transport. In the article, we propose an algorithm and a methodology for solving multidimensional traffic congestion objectives, especially for intersections, based on combining machine learning with the templates method—the golden template algorithm with the multiagent game theory. Intersections are modeled as independent players who had to reach an agreement using Nash negotiation. The obtained results showed that the Nash negotiation with multiagents and the golden template modeling have superior results to the model predictive control (MPC) algorithm, improving travel time, the length of traffic queues, the efficiency of travel flows in an unknown and dynamic environment, and the coordination of the agents’ actions and decision making. The proposed algorithm can be used in planning public passenger transport on alternative routes and in ITS management decision making.

Interest Equilibrium and Path Choice in the Development of Construction Land Decrement: A Theoretical Analysis Based on the Multi-Agent Game Model

High-quality development is a new-era requirement for modern urban construction. The implementation of construction land decrement has become an inevitable choice to solve the urban land use dilemma and achieve high-quality social and economic development. It is a game process for stakeholders. Based on the multi-agent game model, this research analyzes the game relationship and strategy selection between governments, government and farmers, government and developers, and developers and farmers and examines the balance of interests among multiple subjects. From the perspective of space dimension, index dimension, use dimension and time dimension, a game path is constructed for construction land decrement. The research shows the following: There is a clear relationship between the supervision cost of the higher-level government and the intensity of rewards and punishments for the lower-level government and the implementation of the decrement. The district (county) government’s compensation plan and farmers’ perceptions of the success of the boycott will affect the strategic choices of both parties. Governments at all levels play a direct role in restraining development behavior and supervising rent-seeking behavior after developers intervene in volume reduction. It also proposes optimizing the ratio of game factors such as costs, benefits, rewards, verification and punishment to achieve the game equilibrium of the expression of the interests of village residents, developers and governments at all levels. Based on the path planning and coupling of response, the multi-scheme dynamic path selection for the development of construction land decrement is realized.

Multi-Microgrid Energy Management Strategy Based on Multi-Agent Deep Reinforcement Learning with Prioritized Experience Replay

The multi-microgrid (MMG) system has attracted more and more attention due to its low carbon emissions and flexibility. This paper proposes a multi-agent reinforcement learning algorithm for real-time energy management of an MMG. In this problem, the MMG is connected to a distribution network (DN). The distribution network operator (DSO) and each microgrid (MG) are modeled as autonomous agents. Each agent makes decisions to suit its interests based on local information. The decision-making problem of multiple agents is modeled as a Markov game and solved by the prioritized multi-agent deep deterministic policy gradient (PMADDPG), where only local observation is required for each agent to make decisions, the centralized training mechanism is applied to learn coordination strategy, and a prioritized experience replay (PER) strategy is adopted to improve learning efficiency. The proposed method can deal with the non-stationary problems in the process of a multi-agent game with partial observable information. In the execution stage, all trained agents are deployed in a distributed manner and make decisions in real time. Simulation results show that according to the proposed method, the training process of a multi-agent game is accelerated, and multiple agents can make optimal decisions only by local information.

Neuro-adaptive control for searching generalized Nash equilibrium of multi-agent games: A two-stage design approach

This paper investigates the generalized Nash equilibrium searching problem of multi-agent games over weight-unbalanced directed networks. In this problem, the feasible action set of each agent is not only constrained by private convex sets and shared coupling equality, but also constrained by its own uncertain dynamics. Moreover, the local cost function is related to both his own actions and others, and each agent is only allowed access to neighbor information. To address this problem, we propose a neuro-adaptive control strategy based on two-stage design. In the first stage of design process, an approximator with adaptive-gain is designed to generate a virtual trajectory that converges to a necessary Nash equilibrium, the compensator based on consensus-tracking is used to obtain non-neighbor information and several auxiliary state variables are used to exchange information with local neighborhoods on a digraphs to deal with equality constraints. In the second stage of the strategy, we further developed a neuro-adaptive tracking controller based on one-step design for tracking the virtual trajectory. In the controller design, the virtual control variables and the actual control laws are obtained in a collective way, without repeating the design process. We introduce a variable called the minimum learning parameter to reduce the number of online learning parameters of the neural network. By using tools from variational inequality theory and Lyapunov stability theory, we prove that the proposed control strategy can drive all agents reach Nash equilibrium. Finally, the effectiveness of control strategy is verified by simulation example.

Multi-agent game analysis on standardized discretion of environmental administrative penalty

An environmental administrative penalty is a powerful tool to regulate environmental pollution and ecological destruction by punishing intentional violations. Still, unchecked discretion may lead to excessively low or high penalties, breaking our balance of desire for uniformity with the need for discretion. To achieve the balance, regulators may use standardized discretion to achieve greater compliance by punishing intentional violations in a standardized way. However, policymakers and scholars have different attitudes on understanding whether standardized discretion helps enforcement. For this purpose, we construct a muti-agent dynamic game under the standardized discretion of environmental administrative penalty (SDEAP). The results show that: i) SDEAP can positively affect firms’ output and emission reduction efforts but negatively affect environmental quality; ii) The lower limit of SDEAP can positively affect environmental quality but negatively affects firms’ output and emission reduction efforts; iii) The upper limit of SDEAP can positively affect firms’ output, emission reduction efforts, and environmental quality; iv) SDEAP can restrict law enforcement and improve firms’ efforts to reduce emissions. This work can be helpful both to firms and the government as the basis for developing and implementing SDEAP.

Resilient Nash Equilibrium Seeking in Multiagent Games Under False Data Injection Attacks

This article proposes a resilient distributed Nash equilibrium (NE) seeking algorithm for noncooperative games with multiple double-integrator agents who suffer from false data injection (FDI) attacks. A malicious attacker injects false data into agents’ actuators and sensors so that agents’ strategies deviate from the NE of the game with the compromised control inputs and interactive information. First, the robustness of the seeking algorithm against the FDI attacks is analyzed. Then, to mitigate the effect of the attacks on agents’ strategies, the false data injected in the actuators and sensors are regarded as extended states which can be observed by extended state observers (ESOs). Thus, a resilient NE seeking algorithm is proposed based on ESOs. The resilient algorithm can drive the system to converge to the NE without requiring any information about the nature of the attacks. An explicit criterion is given to ensure the effectiveness of the designed algorithms. An example is given to illustrate the results.

Multiagent Game of Intelligent Building Detection and Its Harm Rumor Analysis

The “14th Five‐Year” development plan of the construction industry of the Chinese government makes it clear that intelligent buildings are the general direction of future construction development. However, the occurrence of construction accidents in recent years has brought huge casualties. With the rapid development of third‐party inspection institutions, their judgment has become an important threshold for the access to smart buildings. The failure of inspection institutions to detect the renewal of leases may lead to the failure of intelligent systems, leading to the failure of construction equipment and causing major safety accidents such as the uncontrolled fall of construction elevators and the fall of tower crane items. As the supervisor of third‐party inspection institutions, government functional departments are responsible for the failure of intelligent systems. All of them undertake the important role of intelligent building security. This paper builds an evolutionary game model with government regulatory authorities, third‐party testing institutions, and intelligent construction enterprises as the three game parties, based on variables that affect the factors of the players, and discusses the possible scenarios of the stabilization strategy, which draws the following points: (1) the increase of incentives and penalties by the government is conducive to promoting the construction of high‐quality intelligent buildings and the normative behavior of the third party without intention to seek rent, but the increase of incentives will not be conducive to the government fulfilling its regulatory responsibilities. (2) The reasonable reward and punishment mechanism set by the government must meet the condition that the sum of rewards and punishments for all parties is greater than their speculative returns so as to ensure the safety of intelligent buildings in the evolving and stable market environment. (3) The accountability of the superior government for the dereliction of duty of the regulatory authorities is of great significance to enhancing the stability of enterprises in building high‐quality intelligent buildings. (4) It is also an effective way to avoid the construction of low‐quality smart buildings by improving the listed sales income of enterprises and increasing the rent‐seeking cost of enterprises. Combined with the improved SIR model, it is calculated that after the occurrence of a construction accident, when the rumor first appears, the rumor will quickly spread out of control without timely management and supervision. The conclusion of the game model provides a great reference value for the government’s regulatory behavior. The quality of the government’s regulatory behavior is directly related to the future of high‐quality development of intelligent construction and also expands the application field of evolutionary game theory and SIR theory. The government should actively manage third‐party testing institutions and intelligent building enterprises before construction accidents happen. After the accident, strict management and supervision should be carried out on well‐known bloggers before the emergence of rumors so as to reduce the possibility of standardizing individuals’ reprinting of rumors and avoid the uncontrollable negative impact in the later period, thus seriously affecting the credibility of government supervision.

Neighborhood Cooperative Multiagent Reinforcement Learning for Adaptive Traffic Signal Control in Epidemic Regions

Nowadays, multiagent reinforcement learning (MARL) have shared significant advances in the adaptive traffic signal control (ATSC) problems. For most of the researches, agents are all isomorphic, which disregards the situation in which isomerous intersections cooperative together in a real ATSC scenario, especially in epidemic regions where different intersections have quite different levels of importance. To this end, this paper models the ATSC problem as a networked Markov game (NMG), in which agents take into account information, including traffic conditions of it and its connected neighbors. A cooperative MARL framework named neighborhood cooperative hysteretic DQN (NC-HDQN) is proposed. Specifically, for each NC-HDQN agent in the NMG, first, the framework analyses correlation degrees with their connected neighbors and weighs observations and rewards by these correlations. Second, NC-HDQN agents independently optimize their strategies on the weighted information using hysteretic DQN (HDQN), which is designed to learn optimal joint strategies in cooperative multiagent games. Third, a rule-based NC-HDQN method and a Pearson correlation coefficient based NC-HDQN method, i.e., empirical NC-HDQN (ENC-HDQN) and Pearson NC-HDQN (PNC-HDQN), respectively, are designed. The first method maps the correlation degree between two connected agents according to vehicle numbers on roads between the two agents. In contrast, the second method uses the Pearson correlation coefficient to calculate the correlation degree adaptively. Our methods are empirically evaluated in both a synthetic scenario and two real-world traffic scenarios and give better performances in almost every standard test metric for ATSC.

China’s LNG import risk assessment based on the perspective of global governance

The uncertainty of LNG import risks will have a significant impact on China's energy security. This paper establishes a multi-agent game model based on the current LNG transportation network and global LNG supply and demand pattern, evaluates the LNG import risks faced by China under the global governance model, and simulates and predicts the optimal LNG import strategy of China in 2030. The research results show that between 2007 and 2020, China’s LNG import risks increased rapidly compared with political risks and national risks. From the perspective of risk identification, the comprehensive risk of China's imports from Southeast Asia and Australia is lower. However, due to the increasing demand gap for LNG, the Middle East, and Africa are still necessary supply sources. It is estimated that by 2030, the future LNG market will be oversupplied, and the United States is expected to become the world's top LNG supplier. China will reduce its dependence on Africa and the Middle East, and import a large amount of natural gas from the United States, Australia, Qatar, and Russia to reduce supply risks. From the perspective of import market competition, the new analysis model proposed in this article provides an effective tool for exploring the optimal strategy for LNG import.

Coordinated Control Strategy of Electricity-Heat-Gas Integrated Energy System Considering Renewable Energy Uncertainty and Multi-Agent Mixed Game

The Integrated Energy System (IES) can promote social energy transformation and low-carbon development, which is also an effective means to make contributions to energy structure optimization, energy consumption reduction, and new energy consumption. However, the IES has the characteristics of complex energy flow, and strong uncertainty with multi-agents. Therefore, traditional mathematical optimization models are difficult to comprehensively and accurately reflect the interest needs of different entities in the integrated energy microgrid. Aiming at this problem, a two-level collaborative control strategy model of “electricity-heat-gas” IES based on multi-agent deep reinforcement learning is proposed in this paper. The upper layer of this model is a multi-agent hybrid game decision-making model based on the Multi-Agent Deep Deterministic Policy Gradient algorithm (MADDPG), and the lower layer contains the power and gas flow calculation model. The lower model provides the upper model with the energy flow data of the IES and the upper layer rewards the decision-making behavior of the agent based on the energy flow data provided by the lower layer. Effectively solving the high-dimensional nonlinear optimization problem existing in the complex coupling network, this method can improve the convergence and training speed of the model. In this paper, the IEEE 33-node distribution network and 20-node gas network coupling system are provided to verify the model. The simulation results show that the proposed collaborative control strategy method can provide effective decision-making for electric-agent and gas-agent and realize the efficient and economic operation of the integrated energy system.

Multi-Agent Schedule Optimization Method for Regional Energy Internet Considering the Improved Tiered Reward and Punishment Carbon Trading Model

Regional energy internet (REI) contains massive market agents, whose interests and objectives vary from each other. In consequence, it is challenging to stimulate the energy conservation and emissions reduction participation of each agent by the conventional schedule optimization method. This paper proposes a multi-agent schedule optimization method for REI considering the improved tiered reward and punishment carbon trading model. Firstly, the energy flow constraints and device constraints of REI are established. Secondly, to tighten restrictions on carbon emissions, the relative carbon emission is used as the criterion to establish the improved tied reward and punishment carbon trading model. Next, to analyze the real multi-agent game situation in the market, different agents are classified, and the objective functions are defined based on their revenue. Finally, a two-layer algorithm is used to solve the above multi-agent model. Simulation results verify that the proposed method can effectively reduce carbon emissions and significantly enhance the revenue of the region.

Viscoelastic Fluid-Inspired Swarm Behavior to Reduce Susceptibility to Local Minima: The Chain Siphon Algorithm

We present a novel distributed robotic swarm algorithm inspired by the open channel siphon phenomenon displayed in certain viscoelastic fluids. Self-siphoning viscoelastic fluids are often able to mitigate the trapping effects of local minima in the environment. Using a similar strategy, our algorithm enables a robot swarm to mitigate the trapping effects of local minima in potential fields. Once a robot senses the goal, local communication between robots is used to propagate path-to-goal gradient information through the swarm's communication graph. This information is used to augment each agent's local potential field, reducing the local minima traps and often eliminating them. We perform hardware experiments using the Georgia Tech Miniature Autonomous Blimp (GT-MAB) aerial robotic platforms as well as Monte Carlo simulations conducted in the Simulating Collaborative Robots in Massive Multi-Agent Game Execution (SCRIMMAGE) simulator. We compare the new method to other potential field based swarm behaviors that both do and do not incorporate local minima fixes. The distributed algorithm generates self-siphoning behavior within the robotic swarm, and this reduces its susceptibility to local minima.

Noncooperative multiagent game model for the electricity market on the basis of the renewable energy portfolio standard

AbstractIn China, after the revision of the renewable energy policy, transactions among the main parties involved in the electricity market are becoming increasingly complicated. Nevertheless, the impact of the Renewable Energy Portfolio Standard (RPS) on market equilibrium has not been studied so far. Hence, in this study, a noncooperative multiagent game model for the electricity market was developed on the basis of the recently released RPS and by considering various factors, such as green certificate transactions and the changing of the selling entities. The improved firefly algorithm was used to solve the model, and problematic gaming was analyzed empirically. Once the new RPS is implemented, the price of bound electricity will become higher than that of nonbound electricity, and the trading volume of bound electricity will increase. Because fossil energy producers will no longer be responsible for the electricity quotas, their electricity trading volumes will now be more related to the market demand than they were earlier. Renewable energy companies will increase their revenues by reducing their bound electricity sales. If the RPS increases the electricity quota ratios, the costs incurred by electricity sales companies and large power users will also increase, leading the electricity sales companies to reduce their electricity trading volumes. The study findings will help analyze the impacts of implementing the RPS and provide a basis and a reference for policymakers to make improvements to the standard in the future.

A Multi-Agent Game-Based Incremental Distribution Network Source–Load–Storage Collaborative Planning Method Considering Uncertainties

How to obtain the optimal decision-making scheme based on the investment behavior of various stakeholders is an important issue that needs to be solved urgently in incremental distribution network planning. To this end, this article introduces the virtual player “Nature” to realize the combination of the game theory and robust optimization and proposes an incremental distribution network source–load–storage collaborate planning method with a multi-agent game. First, the planning and decision-making models of a DG investment operator, a distribution network (DN) company, power consumers, and a distributed energy storage (DES) investment operator are constructed, respectively. Then the static game behaviors between the DG investment operator and distribution network company, as well as the DG investment operator and the DES investment operator, are analyzed based on the transfer relations between these four participants. At the same time, robust optimization is used to deal with the uncertainty of the DG output, and the virtual player “Nature” is introduced to study the dynamic game behavior between the DG investment operator and the distribution company. Finally, a dynamic–static joint game planning model is proposed. The simulation results verify the correctness and effectiveness of the proposed method.

Evaluating natural gas supply security in China: An exhaustible resource market equilibrium model

The continuous growth of natural gas demand in China has brought about supply security from the insufficient supply of domestic resources and ever-increasing dependency on natural gas imports. A multi-agent game framework for supply security is proposed to investigate availability, accessibility, and affordability along natural gas industry and supply chain. In this framework, an exhaustible resource market equilibrium model based on mixed complementary programming (MCP) is developed, taking into account the dynamic depletion of the remaining recoverable reserves, to evaluate the gas supply security in China. The results indicate the supply security of China's natural gas is far from optimistic from 2020 to 2049, and the newly-increased recoverable reserves of 200 bcm/a are required to meet the expanding scale of the natural gas market. However, it will face supply risks with lower Reserve-Production ratios after 2030 and higher dependencies after 2040. Therefore, the equilibrium model is used to simulate for seeking feasible solutions to ensure the natural gas supply security from three aspects: market structure, discount rate, and resources tax. Several combinational strategies suitable for China's natural gas supply security are suggested according to the market structure, natural gas import, discount rate, and resources tax for building a more diversification system of natural gas supply, establishing an efficient allocation system of the natural gas market and setting up an effective strategic stockpile system of natural gas resources.