Open-loop Stackelberg Equilibrium Research Articles

Two-player nonlinear Stackelberg differential game via off-policy integral reinforcement learning

For the nonzero-sum games, players have taken different strategies to achieve the Nash equilibrium. However, regarding hierarchical optimization and asymmetric information, Nash equilibrium is ineffective. The Stackelberg game provides us with a leader–follower strategy to cope with this problem. This paper solved two-player continuous-time nonlinear Stackelberg differential games with unknown system dynamics using an off-policy integral reinforcement learning (IRL) technique. A two-level hierarchy optimal control problem is the Stackelberg differential game, and locating the open-loop Stackelberg equilibrium is equivalent to solving the coupled Hamiltonian-Jacobi (HJ) equations. First, the optimal strategies for the leader and the follower are derived. Then, the closed-loop system’s asymptotic stability of optimal control solution is proved. Based on the policy iteration (PI) algorithm, an off-policy IRL algorithm is developed to solve the Stackelberg game for completely unknown system dynamics. The control strategies and cost functions of the leader and follower are approximated using neural networks (NNs). It is demonstrated that NNs weight errors of off-policy algorithm are uniformly ultimately bounded (UUB). Lastly, two simulation examples demonstrate that the proposed learning algorithm is capable of obtaining two-player Stackelberg equilibrium strategies.

A Crash Course in Differential Games and Applications

AbstractThe methodology of differential games is a combination of optimal control theory and game theory. It is the natural framework for economic analysis with strategic interaction and dynamical optimization. The theory gained traction by seminal papers in the early seventies, and it gradually found its way into economics. The purpose of this paper is to make theory and applications of differential games easily accessible by explaining the basics and by developing some characteristic applications. The core of the theory focuses on the open-loop and the multiple Markov-perfect Nash equilibria that use the maximum principle and dynamic programming as the techniques to solve the optimal control problems. The applications are the game of international pollution control and the game of managing a lake, which is an example of an ecological system with tipping points. Finally, it is interesting to note that the discovery of time-inconsistency in the open-loop Stackelberg equilibrium had a huge impact on macroeconomics, since policy making under rational expectations is a Stackelberg differential game.

Linear quadratic leader–follower stochastic differential games for mean-field switching diffusions

In this paper, we consider a linear quadratic (LQ) leader–follower stochastic differential game for regime switching diffusions with mean-field interactions. One of the salient features of this paper is that conditional mean-field terms are included in the state equation and cost functionals. Based on stochastic maximum principles (SMPs), the follower’s problem and the leader’s problem are solved sequentially and an open-loop Stackelberg equilibrium is obtained. Further, with the help of the so-called four-step scheme, the corresponding Hamiltonian systems for the two players are decoupled and then the open-loop Stackelberg equilibrium admits a state feedback representation if some new-type Riccati equations are solvable.

A Differential Game for Optimal Water Price Management

This article studies a leader–follower differential game with a finite horizon, where a single buyer reacts to the selling price set by an agency (water supplier). The Open-Loop Stackelberg equilibrium is calculated, assuming that the user demand is fully satisfied (that is, the interior solution is considered), and the following different tariff schemes are analyzed: linear scheme, increasing block tariff, and convex tariff. Numerical simulations highlight how tariff convexity and seasonality in buyer’s preferences affect water price and demand, and the dynamics of the basin over time. The study shows that synchrony or asynchrony between basin recharge and buyer cyclical demand can dramatically affect the dynamics and basin levels observed at the end of the time period considered. Additionally, the presence of a large number of fluctuations in buyer preferences affects basin fluctuations, while natural recharge may help in maintaining acceptable levels of future water demands.

An addendum to the problem of zero-sum LQ stochastic mean-field dynamic games

In this paper, we first address a linear quadratic mean-field game problem with a leader–follower structure. By adopting a Riccati-type approach, we show how one can obtain a state-feedback representation of the pairs of strategies which achieve an open-loop Stackelberg equilibrium in terms of the global solutions of a system of coupled matrix differential Riccati-type equations. In the second part of this paper, we obtain necessary and sufficient conditions for the solvability of the involved coupled generalized Riccati equations.

Backward Stackelberg Differential Game with Constraints: A Mixed Terminal-Perturbation and Linear-Quadratic Approach

We discuss an open-loop backward Stackelberg differential game involving single leader and single follower. Unlike most Stackelberg game literature, the state to be controlled is characterized by a backward stochastic differential equation (BSDE) for which the terminal- instead initial-condition is specified as a priori; the decisions of leader consist of a static terminal-perturbation and a dynamic linear-quadratic control. In addition, the terminal control is subject to (convex-closed) pointwise and (affine) expectation constraints. Both constraints are arising from real applications such as mathematical finance. For information pattern: the leader announces both terminal and open-loop dynamic decisions at the initial time while takes account the best response of follower. Then, two interrelated optimization problems are sequentially solved by the follower (a backward linear-quadratic (BLQ) problem) and the leader (a mixed terminal-perturbation and backward-forward LQ (BFLQ) problem). Our open-loop Stackelberg equilibrium is represented by some coupled backward-forward stochastic differential equations (BFSDEs) with mixed initial-terminal conditions. Our BFSDEs also involve nonlinear projection operator (due to pointwise constraint) combining with a Karush-Kuhn-Tucker (KKT) system (due to expectation constraint) via Lagrange multiplier. The global solvability of such BFSDEs is also discussed in some nontrivial cases. Our results are applied to one financial example.

Comparative Dynamics and Envelope Theorems of Open-Loop Stackelberg Equilibria in Differential Games

A direct proof of the envelope theorems and intrinsic comparative dynamics of locally differentiable open-loop Stackelberg equilibria (OLSE) is given using an extended primal-dual method. It is shown that the follower’s envelope and comparative dynamics results agree in form with those of any player in an open-loop Nash equilibrium, while those of the leader differ. This difference allows, in principle, an empirical test of the leader–follower role in a differential game. Separability conditions are identified on the instantaneous payoff and transition functions under which, for certain parameters, the intrinsic comparative dynamics of the leader’s time-inconsistent OSLE and those in the corresponding optimal control problem are qualitatively identical. However, similar conditions do not exist for time-consistent OSLE.

Stackelberg stochastic differential game with asymmetric noisy observations

ABSTRACT This paper is concerned with a Stackelberg stochastic differential game with asymmetric noisy observation. In our model, the follower cannot observe the state process directly, but could observe a noisy observation process, while the leader can completely observe the state process. Open-loop Stackelberg equilibrium is considered. The follower first solve a stochastic optimal control problem with partial observation, the maximum principle and verification theorem are obtained. Then the leader turns to solve an optimal control problem for a conditional mean-field forward–backward stochastic differential equation, and both maximum principle and verification theorem are proved. A linear-quadratic Stackelberg stochastic differential game with asymmetric noisy observation is discussed to illustrate the theoretical results in this paper. With the aid of some new Riccati equations, the open-loop Stackelberg equilibrium admits its state estimate feedback representation. Finally, an application to the resource allocation and its numerical simulation are given to show the effectiveness of the proposed results.

Wage and employment determination in a dynamic insider–outsider model

In this paper, I develop a differential insider–outsider game in which a union of corporative incumbents chooses the wage of its members by taking into account the optimal employment policy of a firm that, in turn, is assumed to decide the number of outsiders to hire in a spot labour market. Under the assumption that incumbents cannot be fired and commit themselves to a given path of wages, I demonstrate that such a game displays an open-loop Stackelberg equilibrium in which the initial stock of insiders pins down the trajectories of incumbents, entrants and insider wages. Moreover, resorting to numerical simulations, I show that adjustments towards the steady-state equilibrium occur through asymmetric oscillations that mimic the decline of union membership and union wage premia observed in the US all over the last 20 years. In addition, I show that the model provides a positive relationship between the labour market power of the insider union and the impatience of the firm.

Hierarchical optimal control for input-affine nonlinear systems through the formulation of Stackelberg game

Substantial efforts have been undertaken to explore nonzero-sum differential games. Most of these studies are devoted to devising algorithms to pursue Nash equilibrium, where all players with the same access to information will take policies synchronously. However, when it comes to hierarchical optimization and asymmetric information, Nash equilibrium is ineffective. The Stackelberg game provides us with an idea of leader-follower strategy to cope with this conundrum. The paper investigates hierarchical optimal control for continuous-time two-player input-affine systems characterized by nonlinear dynamics and quadratic cost functions. By introducing new costates, this optimization problem is formulated as a Stackelberg game in conjunction with a parametric optimization problem. Besides, the closed-loop information is available for both players. An adaptive learning algorithm is thus developed to approximately obtain the open-loop Stackelberg equilibrium while ensuring the uniform ultimate bounded stability of this closed-loop system, and two approximators structured by neural networks put this purpose into practice. Finally, two numerical examples illustrate that the proposed methodology can accurately obtain optimal solutions, and a comparative example illustrates its characteristics.

Linear-Quadratic Stackelberg Game for Mean-Field Backward Stochastic Differential System and Application

This paper is concerned with a new kind of Stackelberg differential game of mean-field backward stochastic differential equations (MF-BSDEs). By means of four Riccati equations (REs), the follower first solves a backward mean-field stochastic LQ optimal control problem and gets the corresponding open-loop optimal control with the feedback representation. Then the leader turns to solve an optimization problem for a 1×2 mean-field forward-backward stochastic differential system. In virtue of some high-dimensional and complicated REs, we obtain the open-loop Stackelberg equilibrium, and it admits a state feedback representation. Finally, as applications, a class of stochastic pension fund optimization problems which can be viewed as a special case of our formulation is studied and the open-loop Stackelberg strategy is obtained.

An Open-Loop Stackelberg Strategy for the Linear Quadratic Mean-Field Stochastic Differential Game

This paper is concerned with the open-loop linear–quadratic (LQ) Stackelberg game of the mean-field stochastic systems in finite horizon. By means of two generalized differential Riccati equations, the follower first solves a mean-field stochastic LQ optimal control problem. Then, the leader turns to solve an optimization problem for a linear mean-field forward–backward stochastic differential equation. By introducing new state and costate variables, we present a sufficient condition for the existence and uniqueness of the Stackelberg strategy in terms of the solvability of some Riccati equations and a convexity condition. Furthermore, it is shown that the open-loop Stackelberg equilibrium admits a feedback representation involving the new state and its mean. Finally, two examples are given to show the effectiveness of the proposed results.

A Stackelberg Order Execution Game

We study a sequential order execution differential game over a finite time horizon in the Stackelberg duopoly framework, in contrast to the simultaneous Nash game investigated by Carlin et al (2007). In our game, there are two players, one leader and one follower, who maximize their expected trading payoffs respectively by trading a single risky asset whose price dynamics follows the linear-price market impact model of Almgren & Chriss (2001). We derive a closed-form solution for the unique open-loop Stackelberg equilibrium. We then develop new and complementary managerial insights by looking at both players' equilibrium behavior in terms of trading speeds, inventory positions, price dynamics, and first mover's advantage. We also quantify different behavioral metrics of the same players under our sequential game and the aforementioned simultaneous Nash game of Carlin et al (2007).

Application of Open-Loop Stackelberg Equilibrium to Modeling a Driver's Interaction with Vehicle Active Steering Control in Obstacle Avoidance

The increasing use of active front steering (AFS) technology for obstacle avoidance raises the question of drivers’ interaction with vehicle automation. Mathematical models capable of representing such interaction are in demand for driver behavior study. This paper presents the application of open-loop Stackelberg equilibrium to modeling a driver's interaction with vehicle AFS control in an obstacle avoidance scenario, where both the driver and the AFS controller are exerting steering control to the vehicle. In this paper, such driver–AFS interactive steering control is modeled as a leader–follower game. Mathematical expressions of the driver's and the AFS controller's steering control strategies are derived using the linear quadratic dynamic optimization approach and the distributed model predictive control (DMPC) approach. These two approaches are found to give identical control gains, which suggest their equivalence in representing driver–AFS interaction. The DMPC approach is found to consume far less computation time due to its numerical nature. Mathematical modifications to the steering control strategies are then introduced to allow practical implementation for a future experimental study. Simulation results including time histories of steering angles and vehicle responses are illustrated and discussed.

Leader–follower stochastic differential game with asymmetric information and applications

This paper is concerned with a leader–follower stochastic differential game with asymmetric information, where the information available to the follower is based on some sub-σ-algebra of that available to the leader. Such kind of game problem has wide applications in finance, economics and management engineering such as newsvendor problems, cooperative advertising and pricing problems. Stochastic maximum principles and verification theorems with partial information are obtained, to represent the Stackelberg equilibrium. As applications, a linear–quadratic leader–follower stochastic differential game with asymmetric information is studied. It is shown that the open-loop Stackelberg equilibrium admits a state feedback representation if some system of Riccati equations is solvable.

Pricing, Spectrum Sharing, and Service Selection in Two-Tier Small Cell Networks: A Hierarchical Dynamic Game Approach

Small cells overlaid with macrocells can increase the capacity of two-tier cellular wireless networks by offloading traffic from macrocells. To motivate the small cell service providers (SSPs) to open portion of the access opportunities to macro users (i.e., to operate in a hybrid access mode), we design an incentive mechanism in which the macrocell service provider (MSP) could pay to the SSPs. According to the price offered by the MSP, the SSPs decide on the open access ratio, which is the ratio of shared radio resource for macro users and the total amount of radio resource in a small cell. The users in this two-tier network can make service selection decisions dynamically according to the performance satisfaction level and cost, which again depend on the pricing and spectrum sharing between the MSP and SSPs. To model this dynamic interactive decision problem, we propose a hierarchical dynamic game framework. In the lower level, we formulate an evolutionary game to model and analyze the adaptive service selection of users. An evolutionary stable strategy (ESS) is considered to be the solution of this game. In the upper level, the MSP and SSPs sequentially determine the pricing strategy and the open access ratio, respectively, taking into account the distribution of dynamic service selection at the lower-level evolutionary game. A Stackelberg differential game is formulated where the MSP and SSPs act as the leader and followers, respectively. An open-loop Stackelberg equilibrium is considered to be the solution of this game. We also extend the hierarchical dynamic game framework and investigate the impact of information delays on the equilibrium solutions. Numerical results show the effectiveness and advantages of dynamic control of the open access ratio and pricing.

ε-Subgame Perfectness of an Open-Loop Stackelberg Equilibrium in Linear-State Games

Open-loop Stackelberg equilibria in linear-state games are subgame perfect. This result holds under the hypothesis of unconstrained final state; whereas we need to take into account suitable final-state conditions in order to correctly formalize certain economic problems. A striking contribution of this paper is that it tackles the consistency problem for an open-loop Stackelberg equilibrium in linear-state games with a final-state constraint in the leader’s problem. In this paper, after proving that such a type of equilibrium is not subgame perfect, we introduce a weaker definition of subgame perfectness, which we call e-subgame perfectness. This new definition can be applied to the open-loop Stackelberg equilibrium of a constrained linear-state game. Finally, we present some explanatory examples to show how the definition of e-subgame perfectness can be meaningful.

Cartel-fringe models of the oil market: a quantitative assessment

This paper provides a quantitative analysis of different equilibrium concepts employed in the literature on the cartel-fringe models. It is shown that small changes in parameter values can greatly affect outcomes due to changes in extraction schedules. Furthermore, it is shown that results of the closed-loop Stackelberg equilibrium are often closer to the results of the open-loop Nash equilibrium than to those of the open-loop Stackelberg equilibrium.

Interactions between government and firms: a differential game approach

In this paper we study an extended version of the model described in Gradus (J. Econ. 81:1092–1109, 1989) in order to determine the optimal taxation policy of a government and its effects on the stock of capital goods growth as a result of the activity developed by firms. It is shown that, by introducing a wealth tax, there exists an optimal wealth tax rate for which the open-loop/feedback Nash/Stackelberg equilibria coincide, maximizing the payments for both agents (government and firms), so that the open-loop Stackelberg equilibrium becomes both time consistent and subgame perfect.

A time-consistent open-loop Stackelberg equilibrium of shelf-space allocation

We propose a differential game to study retailer's allocation strategy of shelf-space shares between the manufacturers of two competing brands. Each manufacturer can influence the allocation decision by her advertising spending to improve her brand's goodwill which in turn affects the demand for her product. The game is played à la Stackelberg with the manufacturers as leaders and the retailer as follower. Stackelberg open-loop equilibrium is characterized and shown to be time-consistent.