Markov Game Research Articles

Approximating Nash equilibrium for anti-UAV jamming Markov game using a novel event-triggered multi-agent reinforcement learning

In the downlink communication, it is currently challenging for ground users to cope with the uncertain interference from aerial intelligent jammers. The cooperation and competition between ground users and unmanned aerial vehicle (UAV) jammers leads to a Markov game problem of anti-UAV jamming. Therefore, a model-free method is adopted based on multi-agent reinforcement learning (MARL) to handle the Markov game. However, the benchmark MARL strategies suffer from dimension explosion and local optimal convergence. To solve these issues, a novel event-triggered multi-agent proximal policy optimization algorithm with Beta strategy (ETMAPPO) is proposed in this paper, which aims to reduce the dimension of information transmission and improve the efficiency of policy convergence. In this event-triggering mechanism, agents can learn to obtain appropriate observation in different moment, thereby reducing the transmission of valueless information. Beta operator is used to optimize the action search. It expands the search scope of policy space. Ablation simulations show that the proposed strategy achieves better global benefits with fewer dimension of information than benchmark algorithms. In addition, the convergence performance verifies that the well-trained ETMAPPO has the capability to achieve stable jamming strategies and stable anti-jamming strategies. This approximately constitutes the Nash equilibrium of the anti-jamming Markov game.

Admission-Based Reinforcement-Learning Algorithm in Sequential Social Dilemmas

Recently, the social dilemma problem is no longer limited to unrealistic stateless matrix games but has been extended to temporally and spatially extended Markov games by multi-agent reinforcement learning. Many multi-agent reinforcement-learning algorithms have been proposed to solve sequential social dilemmas. However, most current algorithms focus on cooperation to improve the overall reward while ignoring the equality among agents, which could be improved in terms of practicality. Here, we propose a novel admission-based hierarchical multi-agent reinforcement-learning algorithm to promote cooperation and equality among agents. We extend the give-or-take-some model to Markov games, decompose the fairness of each agent, and propose an Admission reward. For better learning, we design a hierarchy consisting of a high-level policy and multiple low-level policies, where the high-level policy maximizes the Admission reward by choosing different low-level policies to interact with environments. In addition, the learning and execution of policies are realized through a decentralized method. We conduct experiments in multiple sequential social dilemmas environments and show that the Admission algorithm significantly outperforms the baselines, demonstrating that our algorithm can learn cooperation and equality well.

Collaborative Decision-Making Method of Emergency Response for Highway Incidents

With the continuous increase in highway mileage and vehicles in China, highway accidents are also increasing year by year. However, the on-site disposal procedures of highway accidents are complex, which makes it difficult for the emergency department to fully observe the accident scene, resulting in the lack of sufficient communication and cooperation between multiple emergency departments, making the rescue efficiency low and wasting valuable rescue time, and causing unnecessary injury or loss of life due to the lack of timely assistance. Thus, this paper proposes a multi-agent-based collaborative emergency-decision-making algorithm for traffic accident on-site disposal. Firstly, based on the analysis and abstraction of highway surveillance videos obtained from the Shaanxi Provincial Highway Administration, this paper constructs an emergency disposal model based on Petri net to simulate the emergency on-site disposal procedures. After transforming the emergency disposal model into a Markov game model and applying it to the multi-agent deep deterministic strategy gradient (MADDPG) algorithm proposed in this paper, the multiple agents can optimize the emergency-decision-making and on-site disposal procedures through interactive learning with the environment. Finally, the proposed algorithm is compared with the typical algorithm and the actual processing procedure in the simulation experiment of an actual Shaanxi highway traffic accident. The results show that the proposed emergency-decision-making method could greatly improve collaboration efficiency among emergency departments and effectively reduce emergency response time. This algorithm is not only superior to other decision-making algorithms such as genetic algorithm (EA), evolutionary strategy (ES), and deep Q network (DQN), but also reduces the disposal processes by 28%, 28%, and 42%, respectively, compared with the actual disposal process in three emergency disposal cases. In summary, with the continuous development of information technology and highway management systems, the multi-agent-based collaborative emergency-decision-making algorithm will contribute to the actual emergency response process and emergency disposal in the future, improving rescue efficiency and ensuring the safety of individuals.

Robustness and Sample Complexity of Model-Based MARL for General-Sum Markov Games

Multi-agent reinforcement learning (MARL) is often modeled using the framework of Markov games (also called stochastic games or dynamic games). Most of the existing literature on MARL concentrates on zero-sum Markov games but is not applicable to general-sum Markov games. It is known that the best response dynamics in general-sum Markov games are not a contraction. Therefore, different equilibria in general-sum Markov games can have different values. Moreover, the Q-function is not sufficient to completely characterize the equilibrium. Given these challenges, model-based learning is an attractive approach for MARL in general-sum Markov games. In this paper, we investigate the fundamental question of sample complexity for model-based MARL algorithms in general-sum Markov games. We show two results. We first use Hoeffding inequality-based bounds to show that \(\tilde{{\mathcal {O}}}( (1-\gamma )^{-4} \alpha ^{-2})\) samples per state–action pair are sufficient to obtain a \(\alpha \)-approximate Markov perfect equilibrium with high probability, where \(\gamma \) is the discount factor, and the \(\tilde{{\mathcal {O}}}(\cdot )\) notation hides logarithmic terms. We then use Bernstein inequality-based bounds to show that \(\tilde{{\mathcal {O}}}( (1-\gamma )^{-1} \alpha ^{-2} )\) samples are sufficient. To obtain these results, we study the robustness of Markov perfect equilibrium to model approximations. We show that the Markov perfect equilibrium of an approximate (or perturbed) game is always an approximate Markov perfect equilibrium of the original game and provide explicit bounds on the approximation error. We illustrate the results via a numerical example.

Energy-Efficient Multi-UAVs Cooperative Trajectory Optimization for Communication Coverage: An MADRL Approach

Unmanned Aerial Vehicles (UAVs) can be deployed as aerial wireless base stations which dynamically cover the wireless communication networks for Ground Users (GUs). The most challenging problem is how to control multi-UAVs to achieve on-demand coverage of wireless communication networks while maintaining connectivity among them. In this paper, the cooperative trajectory optimization of UAVs is studied to maximize the communication efficiency in the dynamic deployment of UAVs for emergency communication scenarios. We transform the problem into a Markov game problem and propose a distributed trajectory optimization algorithm, Double-Stream Attention multi-agent Actor-Critic (DSAAC), based on Multi-Agent Deep Reinforcement Learning (MADRL). The throughput, safety distance, and power consumption of UAVs are comprehensively taken into account for designing a practical reward function. For complex emergency communication scenarios, we design a double data stream network structure that provides a capacity for the Actor network to process state changes. Thus, UAVs can sense the movement trends of the GUs as well as other UAVs. To establish effective cooperation strategies for UAVs, we develop a hierarchical multi-head attention encoder in the Critic network. This encoder can reduce the redundant information through the attention mechanism, which resolves the problem of the curse of dimensionality as the number of both UAVs and GUs increases. We construct a simulation environment for emergency networks with multi-UAVs and compare the effects of the different numbers of GUs and UAVs on algorithms. The DSAAC algorithm improves communication efficiency by 56.7%, throughput by 71.2%, energy saving by 19.8%, and reduces the number of crashes by 57.7%.

Multi-Defender Strategic Filtering Against Multi Agent Cyber Epidemics on Multi-Environment Model for Smart Grid Protection

The growing cyber space with the developments in cyber network technologies in smart grid (SG) systems has necessitated questioning the reliability of networks and taking precautions against possible cyber threats. For this reason, defensive strategies and approaches against cyber attacks must be improved to sustain secure information flow of the network connections used in electricity generation, transmission, distribution, and consumption. This paper proposes a multi-agent multi environment deep reinforcement learning (MM-DRL) based defender response against cyber epidemics consisting coordinated cyber-attacks (multi-CAs) in the same time frame scheme to sustain security for SG networks. In this regard, the PMU-connected 123-bus system is integrated as a Markov game. MM-DRL approach is implemented for subenvironments of a typical SG system. Multi-CAs game aims to coordinate PMU signals across intersections to improve the network efficiency of a SG. DRL has been applied to data control recently and demonstrated promising performance where each data signal is regarded as an agent. Conversely, multi-CAs are self-renewing emerging causative agent of electricity theft, network disturbances, and data manipulation in SG systems characterized with wide characteristic diversity and rapid evolution. The game results show that the presented request response algorithm is able to minimize system attack damage and maintain protection duties when compared to a benchmark without request response. In addition, the performance of the MM-DRL approach compared to other developed methods is examined.

Multi-agent reinforcement mechanism design for dynamic pricing-based demand response in charging network

Dynamic pricing for the charging network needs to consider the interdependency of charging stations’ decision-making, and acquire user’s demand information and their price sensitivity on demands. Most existing pricing schemes assume users consistently follow stable and inferrable patterns, while it remains crucial to explore how they may be influenced by historic prices in a charging market, their private utility functions, and the best demand in response. To integrate the strategic user-station interactions and uncertainties into the network dynamic pricing, this paper proposes a multi-agent reinforcement mechanism design framework to simultaneously determine the optimal charging prices for multiple charging stations over a period considering power output limits, unexpected arriving requests, and undetermined charging demands of self-interested users who aim to maximize their own utility. Specifically, the station–user interaction is modelled as a mechanism design problem, and station–station cooperation is captured by the Markov game and solved by multi-agent deep deterministic policy gradient. The objective is to maximize the long-term network revenue considering the social welfare of all users. We evaluate our framework through an experimental study, and the results demonstrate that our approach outperforms the non-cooperative deep deterministic policy gradient algorithm and time-of-use pricing scheme.

A Cooperative Charging Control Strategy for Electric Vehicles Based on Multiagent Deep Reinforcement Learning

The growth of electric vehicles (EVs) significantly increases the residential electricity demand and potentially leads to the overload of the transformer in the distribution grid. Aiming to coordinate the charging control of EVs, this article formulates the EVs charging problem as a Markov game with an unknown transition function and proposes a cooperative charging control strategy based on the multiagent deep reinforcement learning. The uncertainties from the dynamic electricity price, non-EV residential load consumption and drivers’ individual behaviors are considered to construct the dynamic charging environment. Each agent contains a collective-policy model and an independent learner. The collective-policy model is introduced to model other agent's behaviors by approximating their power consumption. The independent learner is used to learn the optimal charging strategy by interacting with the environment. The soft-actor-critic framework is adopted to train the independent learner, enabling the proposed method to address the continuous state and action. Agents are trained with only the local observation and approximation, indicating that the proposed approach is fully decentralized and scalable to the problem with multiple agents. Finally, several numerical studies constructed based on the real-world data demonstrate the effectiveness and scalability of the proposed approach.

Collaborative Multiagent Reinforcement Learning Aided Resource Allocation for UAV Anti-Jamming Communication

In this article, we investigate the anti-jamming problem with joint channel and power allocation for unmanned aerial vehicle (UAV) networks. In particular, we focus on avoiding both mutual interference among UAVs and external malicious jamming to maximize the system Quality of Experience (QoE) relevant to the power consumption. To simultaneously capture the competition and coordination among UAVs, we first model the problem as a local interaction Markov game and then prove it as an exact potential game with at least one Nash equilibrium. Next, we propose a collaborative multiagent layered Q learning (MALQL)-based anti-jamming communication algorithm to reduce the high dimensionality of the action space and analyze the asymptotic convergence of the proposed algorithm. Simulation results show the effectiveness of the proposed algorithm, which outperforms the traditional multiagent <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula> learning algorithm when suffering from different jamming strategies.

Neighborhood Cooperative Multiagent Reinforcement Learning for Adaptive Traffic Signal Control in Epidemic Regions

Nowadays, multiagent reinforcement learning (MARL) have shared significant advances in the adaptive traffic signal control (ATSC) problems. For most of the researches, agents are all isomorphic, which disregards the situation in which isomerous intersections cooperative together in a real ATSC scenario, especially in epidemic regions where different intersections have quite different levels of importance. To this end, this paper models the ATSC problem as a networked Markov game (NMG), in which agents take into account information, including traffic conditions of it and its connected neighbors. A cooperative MARL framework named neighborhood cooperative hysteretic DQN (NC-HDQN) is proposed. Specifically, for each NC-HDQN agent in the NMG, first, the framework analyses correlation degrees with their connected neighbors and weighs observations and rewards by these correlations. Second, NC-HDQN agents independently optimize their strategies on the weighted information using hysteretic DQN (HDQN), which is designed to learn optimal joint strategies in cooperative multiagent games. Third, a rule-based NC-HDQN method and a Pearson correlation coefficient based NC-HDQN method, i.e., empirical NC-HDQN (ENC-HDQN) and Pearson NC-HDQN (PNC-HDQN), respectively, are designed. The first method maps the correlation degree between two connected agents according to vehicle numbers on roads between the two agents. In contrast, the second method uses the Pearson correlation coefficient to calculate the correlation degree adaptively. Our methods are empirically evaluated in both a synthetic scenario and two real-world traffic scenarios and give better performances in almost every standard test metric for ATSC.

On the complexity of computing Markov perfect equilibrium in general-sum stochastic games.

Similar to the role of Markov decision processes in reinforcement learning, Markov games (also called stochastic games) lay down the foundation for the study of multi-agent reinforcement learning and sequential agent interactions. We introduce approximate Markov perfect equilibrium as a solution to the computational problem of finite-state stochastic games repeated in the infinite horizon and prove its PPAD-completeness. This solution concept preserves the Markov perfect property and opens up the possibility for the success of multi-agent reinforcement learning algorithms on static two-player games to be extended to multi-agent dynamic games, expanding the reign of the PPAD-complete class.

Multiagent Deep Reinforcement Learning-Aided Output Current Sharing Control for Input-Series Output-Parallel Dual Active Bridge Converter

This letter proposes a multiagent soft actor-critic (MASAC) enabled multiagent deep reinforcement learning (MADRL) algorithm for output current sharing of the input-series output-parallel dual active bridge converter. The modular converter is partitioned into different submodules, which are treated as DRL agents of Markov games. Furthermore, all agents are executed decentralized to provide online control decisions after collaborative training. The proposed MASAC algorithm verified in a 150 V/50 V hardware prototype shows optimal dynamic performance.

Computing Assistance From the Sky: Decentralized Computation Efficiency Optimization for Air-Ground Integrated MEC Networks

This letter proposes a multi-agent deep reinforcement learning (MADRL) framework for resource allocation in air-ground integrated multi-access edge computing (MEC) networks, where unmanned aerial vehicles (UAVs) provide computing services in addition to ground-computing access points (GCAPs). For maximizing the computation efficiency, the resource allocation problem is formulated as the mixed-integer programming problems. Then, we develop a cooperative deep deterministic policy gradient (CODDPG) algorithm to solve the problem via an observable Markov game. The simulation results demonstrate that the proposed algorithm outperforms centralized reinforcement learning in terms of the computation efficiency.

Learning attack-defense response in continuous-time discrete-states Stackelberg Security Markov games

ABSTRACT Researchers have become interested in security games in recent decades as a result of its successful application in real-world security issues. The security model is based on the Stackelberg Security Game (SSG), in which defenders (leaders) select a defensive strategy based on the optimal reaction of attackers (followers), who, at equilibrium, select the predicted assaulting strategy as a response. These applications, on the other hand, do not account for the time constraints posed by the game’s players’ journey time. Furthermore, players should be able to cope with dynamic settings in which their knowledge of the environment changes on a regular basis, allowing them to perform more effectively. This research proposes a security model based on a continuous-time Reinforcement Learning (RL) approach implemented using a temporal difference method that takes prior information into account to address these issues. We use a controlled, ergodic continuous-time Markov game to model the SSG. The game framework model assumes that all information is available. We calculate the number of transitions over a time interval divided by the entire value of the holding time to estimate the transition rates. The arithmetic mean of the observed cost of the individual players is used to estimate the cost for defenders and attackers. An iterated proximal/gradient approach is used to calculate the SSG equilibrium point. We offer a continuous-time random walk method for game implementation. In a numerical case relevant to rain-forest hazards, we analyze the performance of the suggested RL security solution and discuss the problems that should be considered in future.

Multiobjective Edge Server Placement in Mobile-Edge Computing Using a Combination of Multiagent Deep Q-Network and Coral Reefs Optimization

The growth of telecommunication technologies, especially 5G, the growing popularity of smart mobile devices, the emergence of smart cities and Internet of Things (IoT), and the easy use of these equipments have led the cloud users to utilize their various services. Real-time applications and the use of big data have caused cloud service providers (CSPs) to move their servers to the edge of the network and in the vicinity of users to maintain the quality of their services. For this purpose, the concept of mobile-edge computing (MEC) was formed. Applications often have heavy computing complexity on mobile devices or require a lot of data to process. Moreover, in order to save energy consumption of the batteries of this equipment, offloading them on the network resources can transfer the computational complexity from the users’ equipment to the network resources. The resource placement (RP) is one of the major challenges in this area. Improper resource topology upsets their load balancing and increases access latency. In the proposed method of this article, the cellular mobile network is divided into smaller areas and using the coral reefs optimization (CRO) algorithm, the optimal placement of resources in each of these areas will be locally performed. The deep <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula> -network (DQN) and Markov game (MG) are used to optimize global RP to reduce global latency and to improve resource load balancing as its two objectives. The results of the experiments show that the proposed method has significantly improved its objectives and server’s energy efficiency, compared to some similar works in this area.

Energy-efficient personalized thermal comfort control in office buildings based on multi-agent deep reinforcement learning

In a shared office space, the percentage of occupants with satisfied thermal comfort is typically low. The main reason is that heating, ventilation, and air conditioning (HVAC) systems cannot provide individual thermal environment for each occupant within the shared office space. Although personal comfort systems (PCSs) can be adopted to implement heterogeneous thermal environments, they have limited adjustment abilities. At this time, coordinating the operations of PCSs and an HVAC system is a good choice. In this paper, the coordination control problem of PCSs and an HVAC system in a shared office space is investigated to minimize the total energy consumption while maintaining comfortable individual thermal environment for each occupant. Specifically, we first formulate an expected energy consumption minimization problem related to PCSs and an HVAC system. Due to the existence of an inexplicit building thermal dynamics model and uncertain parameters, it is challenging to solve the problem. To overcome the challenge, we reformulate the problem as a Markov game with heterogeneous agents. To promote an efficient cooperation of such agents, we propose a real-time control algorithm based on attention-based multi-agent deep reinforcement learning, which does not require an explicit building thermal dynamics model and any prior knowledge of uncertain parameters. Simulation results based on real-world traces show that the proposed algorithm can reduce energy consumption by 0.7%–4.18% and reduce average thermal comfort deviation by 64.13%–72.08% simultaneously compared with baselines. • A Markov game for coordinating personal comfort systems and HVAC is formulated. • A multi-agent deep reinforcement learning based control method is proposed. • Real-world traces are used to verify the effectiveness of the proposed method.

A Generalized Minimax Q-Learning Algorithm for Two-Player Zero-Sum Stochastic Games

We consider the problem of two-player zero-sum games. This problem is formulated as a min-max Markov game in the literature. The solution of this game, which is the min-max payoff, starting from a given state is called the min-max value of the state. In this work, we compute the solution of the two-player zero-sum game utilizing the technique of successive relaxation that has been successfully applied in the literature to compute a faster value iteration algorithm in the context of Markov Decision Processes. We extend the concept of successive relaxation to the setting of two-player zero-sum games. We show that, under a special structure on the game, this technique facilitates faster computation of the min-max value of the states. We then derive a generalized minimax Q-learning algorithm that computes the optimal policy when the model information is not known. Finally, we prove the convergence of the proposed generalized minimax Q-learning algorithm utilizing stochastic approximation techniques, under an assumption on the boundedness of iterates. Through experiments, we demonstrate the effectiveness of our proposed algorithm.

Reinforcement learning method for target hunting control of multi‐robot systems with obstacles

Aiming at the target encirclement problem of multi-robot systems, a target hunting control method based on reinforcement learning is proposed. First, the Markov game modeling for the multi-robot system is carried out. According to the task of hunting, potential energy models are designed to meet the requirements of arriving at the desired state and avoiding obstacles. The multi-robot reinforcement learning algorithm guided by the potential energy models is presented to perform the hunting, where reinforcement learning principles are combined with the model control. Secondly, based on the potential energy models, the target-tracking hunting strategy and the target-circumnavigation hunting strategy are established. In the former, the consensus tracking of multi-robot systems is achieved by designing the velocity potential energy function. And in the latter, virtual circumnavigation points are added to construct the potential energy function, which realizes the desired circumnavigation. Finally, the effectiveness of target hunting control based on the multi-robot reinforcement learning method is verified by simulation.

Empirical approximation of Nash equilibria in finite Markov games with discounted payoffs

AbstractThis paper deals with finite nonzero‐sum Markov games under a discounted optimality criterion and infinite horizon. The state process evolves according to a stochastic difference equation and depends on players' actions as well as a random disturbance whose distribution is unknown to the players. The actions, the states, and the values of the disturbance are observed by the players, then they use the empirical distribution of the disturbances to estimate the true distribution and make choices based on the available information. In this context, we propose an almost surely convergent procedure—possibly after passing to a subsequence—to approximate Nash equilibria of the Markov game with the true distribution of the random disturbance.

Resource Allocation for Cellular Zero-Touch Deterministic Industrial M2M Networks: A Reinforcement Learning-Based Scheme

In this letter, we investigate the resource allocation problem for cellular zero-touch deterministic industrial machine-to-machine networks. We consider a scenario in which multiple self-adaptive industrial machine-to-machine (M2M) networks are deployed underlaying cellular networks to emulate a variety of vertical use cases in future Industry 4.0 with the objective of guaranteeing the determinacy of latency by learning-inspired resource allocation policy. Specifically, in order to figure out the relationship between latency determinacy and the utilization of wireless resources meanwhile modeling the uncertainty of the stochastic environment, we first formulate the resource allocation problem as a Markov game by jointly considering transmission power control, frequency spectrum allocation, and the selection of base stations. Afterward, a random graph-based sparse long short-term memory network is proposed to solve the optimization problem while reducing the computational complexity. Finally, numerical result demonstrates the effectiveness of the proposed scheme.