Non-cooperative Stochastic Game Research Articles

Reinforcement learning with predefined and inferred reward machines in stochastic games

This paper focuses on Multi-Agent Reinforcement Learning (MARL) in non-cooperative stochastic games, particularly addressing the challenge of task completion characterized by non-Markovian reward functions. We employ Reward Machines (RMs) to incorporate high-level task knowledge. Firstly, we introduce Q-learning with Reward Machines for Stochastic Games (QRM-SG), where RMs are predefined and available to agents. QRM-SG learns each agent’s best-response policy at Nash equilibrium by defining the Q-function in augmented state space that integrates the stochastic game and RM states. The Lemke-Howson method is utilized to compute the best-response policies for the stage game defined by the current Q-functions at each time step. Subsequently, we explore a more challenging scenario where RMs are unavailable and propose Multi-Agent Reinforcement learning with Concurrent High-level knowledge inference (MARCH). MARCH uses automata learning to learn RMs iteratively and combines this process with QRM-SG for learning the best-response policies. The RL episodes where the obtained rewards are inconsistent with the rewards from the current RMs trigger the inference of new RMs. We prove QRM-SG and MARCH converge to the best-response policies under certain conditions. Two scenarios are conducted to demonstrate the superior performance of QRM-SG and MARCH compared to baseline methods.

A Functional Inequality and a New Class of Probabilities in the N-Person Red-and-Black Game

In this paper, we explore a model of an N-player, non-cooperative stochastic game, drawing inspiration from the discrete formulation of the red-and-black gambling problem, as initially introduced by Dubins and Savage in 1965. We extend upon the work of Pontiggia from 2007, presenting a main theorem that broadens the conditions under which bold strategies by all players can achieve a Nash equilibrium. This is obtained through the introduction of a novel functional inequality, which serves as a key analytical tool in our study. This inequality enables us to circumvent the restrictive conditions of super-multiplicativity and super-additivity prevalent in the works of Pontiggia and others. We conclude this paper with a series of illustrative examples that demonstrate the efficacy of our approach, notably highlighting its ability to accommodate a broader spectrum of probability functions than previously recognized in the existing literature.

Stochastic Dynamic Information Flow Tracking game using supervised learning for detecting advanced persistent threats

Advanced persistent threats (APTs) are organized prolonged cyberattacks by sophisticated attackers with the intent of stealing critical information. Although APT activities are stealthy and evade detection by traditional detection tools, they interact with the system components to make progress in the attack. These interactions lead to information flows that are recorded in the form of a system log. Dynamic Information Flow Tracking (DIFT) has been shown to be an effective way to detect APTs using information flows. A DIFT-based detection mechanism dynamically performs security analysis on the information flows to detect possible attacks. However, wide range security analysis using DIFT results in a significant increase in performance overhead and high rates of false-positives and false-negatives. In this paper, we model the strategic interaction between APT and DIFT as a non-cooperative stochastic game. The game unfolds on a state space constructed from an information flow graph (IFG) that is extracted from the system log. The objective of the APT in the game is to choose transitions in the IFG to find an optimal path in the IFG from an entry point of the attack to an attack target. On the other hand, the objective of DIFT is to dynamically select nodes in the IFG to perform security analysis for detecting APT. Our game model has imperfect information as the players are unaware of the actions of the opponent. We consider two scenarios of the game (i) the false-positive and false-negative rates of DIFT (i.e., transition probabilities of the game) are known and (ii) the false-positive and false-negative rates are unknown. For case (i), we propose a value iteration-based algorithm and prove that the solution converges to the optimal solution (Nash equilibrium). Case (ii) translates to an incomplete information game with unknown transition probabilities. For case (ii), we propose a supervised learning-based algorithm, referred to as Hierarchical Supervised Learning (HSL) algorithm. HSL integrates a neural network, to predict the value vector of the game, with a policy iteration algorithm to compute an approximate equilibrium. We implemented our algorithms for cases (i) and (ii) on real attack datasets for nation state and ransomware attacks and validated the performance of our approach. We compared the performance of the HSL algorithm when the transition probabilities are unknown with instances with known transition probabilities and demonstrated that HSL algorithm converges to a solution close to optimal (i.e., optimal value vector) while the value vector obtained using greedy does not converge to optimal for 44.4% of the states and the mean absolute error is almost 200 times that of the HSL.

Multiagent Meta-Reinforcement Learning for Optimized Task Scheduling in Heterogeneous Edge Computing Systems

Mobile edge computing (MEC) brings the potential to address the ever increasing computation demands from the mobile users (MUs). In addition to local processing, the resource-constrained MUs in an MEC system can also offload computation to the nearby servers for remote execution. With the explosive growth of mobile devices, computation offloading faces the challenge of spectrum congestion, which in turn deteriorates the overall quality of computation experience. This paper hence investigates computation task scheduling in a heterogeneous cellular and WiFi MEC system. Such a system provides both licensed and unlicensed spectrum opportunities. Due to the sharing of communication and computation resources as well as the uncertainties, we formulate the problem of computation task scheduling among the competing MUs in a stationary heterogeneous edge computing system as a non-cooperative stochastic game. We propose an approximation-based multi-agent Markov decision process without the global system state observations, under which a multi-agent proximal policy optimization (PPO) algorithm is derived to solve the corresponding Nash equilibrium. When expanding to a non-stationary heterogeneous edge computing system, the obtained algorithm suffers from the slow convergence due to constrained adaptability. Accordingly, we explore meta-learning and propose a multi-agent meta-PPO algorithm, which rapidly adapts the control policy learning to the non-stationarity. Numerical experiments demonstrate performance gains from our proposed algorithms.

Stochastic non‐cooperative game for safety control of shared control vehicle in uncertain environments

AbstractThere are great challenges in the driving of human–machine shared control vehicles (HSCVs) in uncertain environments. Aiming at solving the collision avoidance problem of HSCV considering the uncertainty of surrounding vehicles (SVs), a human–machine cooperative control method for collision avoidance based on stochastic non‐cooperative game (SNCG) is proposed. This method realizes the safe driving of HSCV combined with the prediction of surrounding environment and constructs the interaction model between a driver and an automation system based on the game theory. Firstly, a vehicle–driver coupling model with driver's lateral and longitudinal operation characteristics is established. Next, utilizing the prediction equation, the prediction of the SV motion considering uncertainty is realized. Then, the safety constraints of HSCV are established by considering the chance constraints and the saturation characteristics of the actuators. Finally, based on the distributed stochastic model predictive control (DSMPC) method, the coupled optimization problem with chance constraints is constructed. To solve the coupled optimization problem with chance constraints, the idea of Nash equilibrium solution and the method of chance constraints tightening is adopted. The simulation results illustrate that the proposed method ensures the safe driving of HSCV under uncertain environments and realize the cooperative control between the automation system and the driver.

Two-stage distributionally robust noncooperative games: Existence of Nash equilibrium and its application to Cournot–Nash competition

Two-stage distributionally robust stochastic noncooperative games with continuous decision variables are studied. In such games, each player solves a two-stage distributionally robust optimization problem depending on the decisions of the other players. Existing studies in this area have been limited with strict assumptions, such as linear decision rules, and supposed that each player solves a two-stage linear distributionally robust optimization with a specifically structured ambiguity set. This limitation motivated us to generalize and analyze the game in a nonlinear case. The contributions of this study are (ⅰ) demonstrating the conditions for the existence of two-stage Nash equilibria under convexity and compactness assumptions, and (ⅱ) consideration of a two-stage distributionally robust Cournot–Nash competition as an application, as well as an investigation into the conditions for the existence of market equilibria in an economic sense. We also report some results of numerical experiments to illustrate how distributional robustness affects the decision of each player in the Cournot–Nash competition.

Security investment of interdependent and identical CPSs: Stochastic games with asymmetric information

This paper considers remote state estimation in cyber-physical systems (CPSs) with multiple sensors, where measurements of each sensor are transmitted to the corresponding remote estimator over a shared communication network with interdependent security. A stochastic non-cooperative game framework with asymmetric information is provided, in which each sensor is in pursuit of minimizing the security investment cost on the first stage associated with the expected error covariance as small as possible at the corresponding remote estimator on the second stage. The asymmetry of information among sensors poses a challenge to characterize or compute the Nash equilibria (NE). To overcome the challenge, based on the common information among sensors, the game with asymmetric information is transformed into another game with symmetric information such that a subclass of NE refer to common information based Markov perfect equilibria of the original game can be achieved by using a backward induction algorithm. Finally, a numerical example is presented to verify the obtained results.

On a Simplified Method of Defining Characteristic Function in Stochastic Games

In the paper, we propose a new method of constructing cooperative stochastic game in the form of characteristic function when initially non-cooperative stochastic game is given. The set of states and the set of actions for any player is finite. The construction of the characteristic function is based on a calculation of the maximin values of zero-sum games between a coalition and its anti-coalition for each state of the game. The proposed characteristic function has some advantages in comparison with previously defined characteristic functions for stochastic games. In particular, the advantages include computation simplicity and strong subgame consistency of the core calculated with the values of the new characteristic function.

Distributed Deep Reinforcement Learning for Intelligent Load Scheduling in Residential Smart Grids

The power consumption of households has been constantly growing over the years. To cope with this growth, intelligent management of the consumption profile of the households is necessary, such that the households can save the electricity bills, and the stress to the power grid during peak hours can be reduced. However, implementing such a method is challenging due to the existence of randomness in the electricity price and the consumption of the appliances. To address this challenge, in this article, we employ a model-free method for the households, which works with limited information about the uncertain factors. More specifically, the interactions between households and the power grid can be modeled as a noncooperative stochastic game, where the electricity price is viewed as a stochastic variable. To search for the Nash equilibrium (NE) of the game, we adopt a method based on distributed deep reinforcement learning. Also, the proposed method can preserve the privacy of the households. We then utilize real-world data from Pecan Street Inc., which contains the power consumption profile of more than 1000 households, to evaluate the performance of the proposed method. In average, the results reveal that we can achieve around 12% reduction on peak-to-average ratio and 11% reduction on load variance. With this approach, the operation cost of the power grid and the electricity cost of the households can be reduced.

New inequalities for probability functions in the two-person red-and-black game

We discuss a model of a two-person, non-cooperative stochastic game, inspired by the discrete version of the red-and-black gambling problem presented by Dubins and Savage. Assume that two players hold certain amounts of money. At each stage of the game they simultaneously bid some part of their current fortune and the probability of winning or loosing depends on their bids. In many models of the red-and-black game it is assumed that the win probability is a function of the quotient of the bid of the first player and the sum of both bids. In the literature some additional properties, like concavity or super-multiplicativity are assumed in order to ensure that the bold and timid strategies form a Nash equilibrium, which turns out to be unique in several cases. Our aim is to provide a generalization in which the probability of winning is a two-variable function which depends on both bids. We introduce two new functional inequalities whose solutions lead to win probability functions for which a Nash equilibrium is realized by the bold-timid strategy. Since both inequalities may appear as not quite intuitive, we discuss them in a separate section of the paper and we provide some illustrating examples.

Resource Allocation and Power Control Policy for Device-to-Device Communication Using Multi-Agent Reinforcement Learning

Device-to-Device (D2D) communication is a promising technology that can reduce the burden on cellular networks while increasing network capacity. In this paper, we focus on the channel resource allocation and power control to improve the system resource utilization and network throughput. Firstly, we treat each D2D pair as an independent agent. Each agent makes decisions based on the local channel states information observed by itself. The multi-agent Reinforcement Learning (RL) algorithm is proposed for our multi-user system. We assume that the D2D pair do not possess any information on the availability and quality of the resource block to be selected, so the problem is modeled as a stochastic non-cooperative game. Hence, each agent becomes a player and they make decisions together to achieve global optimization. Thereby, the multi-agent Q-learning algorithm based on game theory is established. Secondly, in order to accelerate the convergence rate of multi-agent Q-learning, we consider a power allocation strategy based on Fuzzy Cmeans (FCM) algorithm. The strategy firstly groups the D2D users by FCM, and treats each group as an agent, and then performs multi-agent Q-learning algorithm to determine the power for each group of D2D users. The simulation results show that the Q-learning algorithm based on multi-agent can improve the throughput of the system. In particular, FCM can greatly speed up the convergence of the multi-agent Q-learning algorithm while improving system throughput.

Stochastic differential games: the potential approach

A noncooperative stochastic differential game (SDG) is said to be a potential game if there exists an associated stochastic control problem whose optimal solutions are Nash equilibria for the given game. In this paper, we give sufficient conditions for an SDG to be a potential game. To this end, our main tools are suitable versions of the stochastic maximum principle. Our results are illustrated with several examples, including team games and separable games, which have Pareto-optimal Nash equilibria.

Convergence Analysis of Sample Average Approximation of Two-Stage Stochastic Generalized Equations

A solution of two-stage stochastic generalized equations is a pair: a first stage solution which is independent of realization of the random data and a second stage solution which is a function of random variables.This paper studies convergence of the sample average approximation of two-stage stochastic nonlinear generalized equations. In particular an exponential rate of the convergence is shown by using the perturbed partial linearization of functions. Moreover, sufficient conditions for the existence, uniqueness, continuity and regularity of solutions of two-stage stochastic generalized equations are presented under an assumption of monotonicity of the involved functions. These theoretical results are given without assuming relatively complete recourse, and are illustrated by two-stage stochastic non-cooperative games of two players.

Incorporating risk preferences in stochastic noncooperative games

ABSTRACTTraditional game-theoretic models of competition with uncertainty often ignore preferences and attitudes toward risk by assuming that players are risk neutral. In this article, we begin by considering how a comprehensive analysis and incorporation of expected utility theory affect players’ equilibrium behavior in a simple, single-period, sequential stochastic game. Although the literature posits that the more risk averse a first mover is, the more likely she is to compete and defend her position as the “leader”, and that the more risk seeking a “follower” is, the more likely he is willing to participate and compete, we find that this behavior may not always be true in this more general setting. Under simple assumptions on the utility function, we perform sensitivity analyses on the parameters and show which behavior changes when deviations from risk neutrality are introduced into a model. We also provide some insights on how risk preferences influence pre-emption and interdiction by looking at how these preferences affect the first mover’s advantage in a sequential setting. This article generates novel insights when a confluence of factors leads players to deviate or change their behavior in many risk analysis settings where stochastic games are used.

Balancing Delay and Energy Efficiency in Energy Harvesting Cognitive Radio Networks: A Stochastic Stackelberg Game Approach

In this paper, we present a cross-layer perspective on data transmission in energy harvesting cognitive radio networks. The physical layer power allocation and network layer delay are jointly considered. The delay optimal power allocation is studied under the intermittence of harvested energy, the randomness of data generation, the variation of channel state, and the uncertainty of the grid price. The signal-interference-ratio constraint is considered to guarantee primary user (PU)’s transmission. The single PU scenario is first investigated. We formulate a stochastic Stackelberg game in bilevel manner. The Nash equilibrium of lower level noncooperative stochastic game is studied (e.g., existence, uniqueness, and solving algorithm) as a basis. Thereafter, the whole stochastic Stackelberg game is analyzed in two circumstances. When the secondary user’s (SUs) information is available at the PU, the Stackelberg equilibrium can be obtained by solving a stochastic mathematical program with complementarity constraints problem; once SUs’ information is unavailable, we design a distributed iterative algorithm and a distributive online algorithm. The multi-PU scenario is discussed, and the corresponding iterative algorithm is given. Finally, simulations are carried out to verify the correctness and demonstrate the effectiveness of proposed algorithms.

An Autonomous Learning-Based Algorithm for Joint Channel and Power Level Selection by D2D Pairs in Heterogeneous Cellular Networks

We study the problem of autonomous operation of the device-to-device (D2D) pairs in a heterogeneous cellular network with multiple base stations (BSs). The spectrum bands of the BSs (that may overlap with each other) comprise the sets of orthogonal wireless channels. We consider the following spectrum usage scenarios: 1) the D2D pairs transmit over the dedicated frequency bands and 2) the D2D pairs operate on the shared cellular/D2D channels. The goal of each device pair is to jointly select the wireless channel and power level to maximize its reward, defined as the difference between the achieved throughput and the cost of power consumption, constrained by its minimum tolerable signal-to-interference-plus-noise ratio requirements. We formulate this problem as a stochastic non-cooperative game with multiple players (D2D pairs) where each player becomes a learning agent whose task is to learn its best strategy (based on the locally observed information) and develop a fully autonomous multi-agent Q-learning algorithm converging to a mixed-strategy Nash equilibrium. The proposed learning method is implemented in a long term evolution-advanced network and evaluated via the OPNET-based simulations. The algorithm shows relatively fast convergence and near-optimal performance after a small number of iterations.

A Stochastic Differential Game Theoretic Study of Multipath Routing in Heterogeneous Wireless Networks

In heterogeneous wireless networks (HWNs), paths constituting multipath routing are characterized by selfish rationality. Each path's intentions of pursuing individual profits may cause unreasonable competition for limited wireless resources, which leads to the unreliable data transport problem. Therefore, multipath routing optimization is a challenge issue in HWNs. This paper provides a novel approach to study this issue by employing game theory. By taking the utility maximization as the design goal, limited bandwidth resources as the constraint and path reliability as the key metric, a noncooperative stochastic differential game model is constructed for HWNs. With the feedback Nash equilibrium solution, an optimal multipath routing strategy is obtained. Theoretical derivations and simulation results verify the validity of the method present in this paper.

A Novel Model for Competition and Cooperation among Cloud Providers

Having received significant attention in the industry, the cloud market is nowadays fiercely competitive with many cloud providers. On one hand, cloud providers compete against each other for both existing and new cloud users. To keep existing users and attract newcomers, it is crucial for each provider to offer an optimal price policy which maximizes the final revenue and improves the competitive advantage. The competition among providers leads to the evolution of the market and dynamic resource prices over time. On the other hand, cloud providers may cooperate with each other to improve their final revenue. Based on a service level agreement, a provider can outsource its users’ resource requests to its partner to reduce the operation cost and thereby improve the final revenue. This leads to the problem of determining the cooperating parties in a cooperative environment. This paper tackles these two issues of the current cloud market. First, we solve the problem of competition among providers and propose a dynamic price policy. We employ a discrete choice model to describe the user’s choice behavior based on his obtained benefit value. The choice model is used to derive the probability of a user choosing to be served by a certain provider. The competition among providers is formulated as a non-cooperative stochastic game where the players are providers who act by proposing the price policy simultaneously. The game is modelled as a Markov Decision Process whose solution is a Markov Perfect Equilibrium. Then, we address the cooperation among providers by presenting a novel algorithm for determining a cooperation strategy that tells providers whether to satisfy users’ resource requests locally or outsource them to a certain provider. The algorithm yields the optimal cooperation structure from which no provider unilaterally deviates to gain more revenue. Numerical simulations are carried out to evaluate the performance of the proposed models.

Dynamic Potential Games: The Discrete-Time Stochastic Case

This paper concerns a class of nonstationary discrete-time stochastic noncooperative games. Our goals are threefold. First, we give conditions to find Nash equilibria by means of the Euler equation approach. Second, we identify subclasses of dynamic potential games. Finally, within one of this subclasses, we identify a further subclass for which Nash equilibria are also Pareto (or cooperative) solutions.

Service rate control of closed Jackson networks from game theoretic perspective

Game theoretic analysis of queueing systems is an important research direction of queueing theory. In this paper, we study the service rate control problem of closed Jackson networks from a game theoretic perspective. The payoff function consists of a holding cost and an operating cost. Each server optimizes its service rate control strategy to maximize its own average payoff. We formulate this problem as a non-cooperative stochastic game with multiple players. By utilizing the problem structure of closed Jackson networks, we derive a difference equation which quantifies the performance difference under any two different strategies. We prove that no matter what strategies the other servers adopt, the best response of a server is to choose its service rates on the boundary. Thus, we can limit the search of equilibrium strategy profiles from a multidimensional continuous polyhedron to the set of its vertex. We further develop an iterative algorithm to find the Nash equilibrium. Moreover, we derive the social optimum of this problem, which is compared with the equilibrium using the price of anarchy. The bounds of the price of anarchy of this problem are also obtained. Finally, simulation experiments are conducted to demonstrate the main idea of this paper.