Equilibrium Policy Research Articles

Relaxed Equilibria for Time-Inconsistent Markov Decision Processes

This paper considers an infinite-horizon Markov decision process (MDP) that allows for general nonexponential discount functions in both discrete and continuous time. Because of the inherent time inconsistency, we look for a randomized equilibrium policy (i.e., relaxed equilibrium) in an intrapersonal game between an agent’s current and future selves. When we modify the MDP by entropy regularization, a relaxed equilibrium is shown to exist by a nontrivial entropy estimate. As the degree of regularization diminishes, the entropy-regularized MDPs approximate the original MDP, which gives the general existence of a relaxed equilibrium in the limit by weak convergence arguments. As opposed to prior studies that consider only deterministic policies, our existence of an equilibrium does not require any convexity (or concavity) of the controlled transition probabilities and reward function. Interestingly, this benefit of considering randomized policies is unique to the time-inconsistent case.

Stochastic Differential Games and a Unified Forward–Backward Coupled Stochastic Partial Differential Equation with Lévy Jumps

We establish a relationship between stochastic differential games (SDGs) and a unified forward–backward coupled stochastic partial differential equation (SPDE) with discontinuous Lévy Jumps. The SDGs have q players and are driven by a general-dimensional vector Lévy process. By establishing a vector-form Ito-Ventzell formula and a 4-tuple vector-field solution to the unified SPDE, we obtain a Pareto optimal Nash equilibrium policy process or a saddle point policy process to the SDG in a non-zero-sum or zero-sum sense. The unified SPDE is in both a general-dimensional vector form and forward–backward coupling manner. The partial differential operators in its drift, diffusion, and jump coefficients are in time-variable and position parameters over a domain. Since the unified SPDE is of general nonlinearity and a general high order, we extend our recent study from the existing Brownian motion (BM)-driven backward case to a general Lévy-driven forward–backward coupled case. In doing so, we construct a new topological space to support the proof of the existence and uniqueness of an adapted solution of the unified SPDE, which is in a 4-tuple strong sense. The construction of the topological space is through constructing a set of topological spaces associated with a set of exponents {γ1,γ2,…} under a set of general localized conditions, which is significantly different from the construction of the single exponent case. Furthermore, due to the coupling from the forward SPDE and the involvement of the discontinuous Lévy jumps, our study is also significantly different from the BM-driven backward case. The coupling between forward and backward SPDEs essentially corresponds to the interaction between noise encoding and noise decoding in the current hot diffusion transformer model for generative AI.

Securing demand–response in smart grids against false pricing attacks

Two-way communication systems in smart grids help to engage consumers in demand–response programs, which can bring many benefits but also make smart grids vulnerable to cyber attacks. In this paper, a cyber attack which aims to disturb the demand–response process by injecting false electricity price signals is considered. A real-time pricing model where the operator has incomplete knowledge of the private demand–response behaviors of the customers is proposed. The vulnerability of the power system to false pricing attacks is analyzed by a Markov decision process, and the dynamic interaction between the attacker and the defender is modeled as a zero-sum Markov game where neither player has full information of the game model. For the solution of the Markov game, a model-free multi-agent reinforcement learning method is proposed to find the Nash Equilibrium policies for both players. The proposed method is applied to the IEEE 34 Node Test Feeder, in which the effect of the defense on mitigating the impact of the attack is demonstrated and different policies of the players given various resources are analyzed. The results shows that the studied cyber attack can cause a maximum of 1.96% unsatisfied load and the proposed defense measure can reduce the unsatisfied load by 50%–70% compared with the cases without defense measures.

On some properties of limited move stability, generalized metarationality, and policy equilibrium in bilateral conflicts

In the Graph Model for Conflict Resolution (GMCR), conflicts are analyzed as in a chess game foreseeing a sequence of moves and countermoves, where the length of this sequence is known as the conflict horizon. In the literature, the stability concepts mostly used are with a fixed horizon, but there are concepts that accommodate a variable horizon. Although these latter concepts have a higher flexibility, they are not applied very frequently perhaps because their properties are not so well understood. In this article, our objective is to shed some light in three stability concepts with variable horizon: limited move, generalized metarationality stabilities and policy equilibrium in bilateral conflicts. In particular, we present some existing inaccuracies in the literature of the GMCR, regarding these concepts. First, we observed that the state anticipated by the focal decision maker (DM) in the limited move stability is not necessarily unique when both the focal DM and his/her opponent move seeking to maximize their respective gains and the focal DM knows his/her opponent’s preferences. Next, we present a counterexample to a result in the literature that relates generalized metarationalities and policy equilibria. Finally, we present a formal proof that Lh implies the MRh stability, and show that the existing justification for this result in the GMCR literature has a problem.

Impact of tariff liberalization on economic and social benefits: Computable general equilibrium application to Kenya

AbstractThis study employed computable general equilibrium policy simulations to examine the impact of import tariff liberalization on socioeconomic outcomes including living standards, cost of living, gross domestic product (GDP) from expenditure, total investment expenditure, intermediate input demand, output, value‐added, tariff revenue, sales tax revenue, indirect tax revenue, direct income tax revenue, factor income tax revenue, and factor demand. Findings from the simulations have policy implications touching on a need to embrace reciprocal tariff liberalization under agreements like strategic trade and investment partnerships and economic partnership agreements as they are associated with welfare gains, reduction in cost of living, and GDP growth. Targeted policy incentives could be directed to specific domestic sectors since tariff liberalization has negative impacts on investment expenditure. Incentives regarding intermediate inputs should be directed to the manufacturing and services sectors for liberalization of agrifood commodities. Targeted policy incentives should be directed to the manufacturing and services sectors which experience decrease in output with tariff reductions. Liberalization of manufactured commodities should be accompanied by value‐added incentives directed to the agrifood and home production–home consumption sectors. For the liberalization of manufactured commodities, targeted policy incentives should be directed to the services sector which experiences job losses while for the liberalization of tariffs on imports of agrifood commodities, targeted policy incentives should be directed to the agrifood and the home production–home consumption sectors, which also lose jobs. These targeted policy incentives could support industrial activity and compensate for revenue loss, especially from customs duties.Related ArticlesGruber, Lloyd. 2013. “Trade, Growth, Poverty, and Politics: Toward a Unified Theory.” Politics & Policy 41(5): 723–64. https://doi.org/10.1111/polp.12034.Onyango, Gedion, and Japheth Otieno Ondiek. 2022. “Open Innovation During the COVID‐19 Pandemic Policy Responses in South Africa and Kenya.” Politics & Policy 50(5): 1008–31. https://doi.org/10.1111/polp.12490.Soto, Gloria. 2012. “Environmental Impact of Agricultural Trade Liberalization under NAFTA.” Politics & Policy 40(3): 471–91. https://doi.org/10.1111/j.1747‐1346.2012.00354.x.

Strategic trade policy in general oligopolistic equilibrium: The case of import tariffs

In a two-country model of general oligopolistic equilibrium with technologically heterogeneous sectors, I study how, in a home-market scenario, a unilateral rise in uniform cross-sector import tariffs affects wages, countrywide profits, and welfare. Firms face resource constraints and wages are simultaneously determined. Economy-wide protectionism reduces the foreign wage without affecting the domestic one. Domestic countrywide profits benefit from a small rise in uniform tariffs, whereas the foreign counterpart is damaged. Domestic welfare is unambiguously hindered. Hence, the general-equilibrium cross-sector perspective goes against the textbook version theory of the optimal tariff in partial equilibrium. Rationalization of these effects suggests a political-economy view on tariff formation in general equilibrium. Then I extend the model to segmented markets, considering uniform specific and ad valorem tariffs for bilateral and unilateral trade policies.

An optimal Bayesian intervention policy in response to unknown dynamic cell stimuli

Interventions in gene regulatory networks (GRNs) aim to restore normal functions of cells experiencing abnormal behavior, such as uncontrolled cell proliferation. The dynamic, uncertain, and complex nature of cellular processes poses significant challenges in determining the best interventions. Most existing intervention methods assume that cells are unresponsive to therapies, resulting in stationary and deterministic intervention solutions. However, cells in unhealthy conditions can dynamically respond to therapies through internal stimuli, leading to the recurrence of undesirable conditions. This paper proposes a Bayesian intervention policy that adaptively responds to cell dynamic responses according to the latest available information. The GRNs are modeled using a Boolean network with perturbation (BNp), and the fight between the cell and intervention is modeled as a two-player zero-sum game. Assuming an incomplete knowledge of cell stimuli, a recursive approach is developed to keep track of the posterior distribution of cell responses. The proposed Bayesian intervention policy takes action according to the posterior distribution and a set of Nash equilibrium policies associated with all possible cell responses. Analytical results demonstrate the superiority of the proposed intervention policy against several existing intervention techniques. Meanwhile, the performance of the proposed policy is investigated through comprehensive numerical experiments using the p53-MDM2 negative feedback loop regulatory network and melanoma network. The results demonstrate the empirical convergence of the proposed policy to the optimal Nash equilibrium policy.

Learning Stationary Nash Equilibrium Policies in \(n\)-Player Stochastic Games with Independent Chains

Learning Stationary Nash Equilibrium Policies in \(n\)-Player Stochastic Games with Independent Chains

Naïve Markowitz policies

AbstractWe study a continuous‐time Markowitz mean–variance portfolio selection model in which a naïve agent, unaware of the underlying time‐inconsistency, continuously reoptimizes over time. We define the resulting naïve policies through the limit of discretely naïve policies that are committed only in very small time intervals, and derive them analytically and explicitly. We compare naïve policies with pre‐committed optimal policies and with consistent planners' equilibrium policies in a Black–Scholes market, and find that the former achieve higher expected terminal returns than originally planned yet are mean–variance inefficient when the risk aversion level is sufficiently small, and always take strictly riskier exposure than equilibrium policies. We finally define an efficiency ratio for comparing return–risk tradeoff with the same original level of risk aversion, and show that naïve policies are always strictly less efficient than pre‐committed and equilibrium policies.

Mortgage interest deductions? Not a bad idea after all

Mortgage interest deductions and other homeownership subsidies are widely believed to be harmful because they redistribute resources from lower-income renters to higher-income homeowners. We argue that renters actually benefit from these policies in general equilibrium for two reasons. First, the rental supply curve is relatively inelastic, which means that rents fall when these policies reduce rental demand. Second, many renters spend most of their income on housing, and these renters gain substantially from rent decreases. We calibrate a quantitative model to match empirical evidence on these factors and show they are strong enough that subsidizing homeownership actually increases welfare.

Identity concealment games: How I learned to stop revealing and love the coincidences

In an adversarial environment, a hostile player performing a task may behave like a non-hostile one in order not to reveal its identity to an opponent. To model such a scenario, we define identity concealment games: zero-sum stochastic reachability games with a zero-sum objective of identity concealment. To measure the identity concealment of the player, we introduce the notion of an average player. The average player’s policy represents the expected behavior of a non-hostile player. We show that there exists an equilibrium policy pair for every identity concealment game and give the optimality equations to synthesize an equilibrium policy pair. If the player’s opponent follows a non-equilibrium policy, the player can hide its identity better. For this reason, we study how the hostile player may learn the opponent’s policy. Since learning via exploration policies would quickly reveal the hostile player’s identity to the opponent, we consider the problem of learning a near-optimal policy for the hostile player using the game runs collected under the average player’s policy. Consequently, we propose an algorithm that provably learns a near-optimal policy and give an upper bound on the number of sample runs to be collected.

Дифференциальная игра «преследование-уклонение» на основе обучения с подкреплением

The paper discusses optimal control algorithms based on actor/critic reinforcement learning (RL) schemes. The algorithms are used to solve pursuit-evasion (PE) problems of differential games. The work focuses on the implementation of agent policy decisions in accordance with the concept of adaptive dynamic programming. The essence of solving the PE game problem is to obtain the control policy of each agent (pursuer and evader) on both sides of the game. The paper proposes an adaptive dynamic programming (ADP) method for solving Nash equilibrium policies in differential pursuit-evasion games for two players. The cost function approximation method is used to calculate the parameters of a neural network (NN) without directly solving the Hamilton-Jacoby equation.

Receding Horizon Games for Modeling Competitive Supply Chains⋆

The vast majority of products we use daily are supplied to us through complex global supply chains that transform raw materials into finished goods and distribute them to end consumers. This paper proposes a modeling methodology for dynamic competitive supply chains based on game theory and model predictive control. We model each manufacturer in the supply chain as a rational utility maximizing agent that selects their actions by finding an open-loop generalized Nash equilibrium of a multi-stage game. To react to competitors and the state of the market, every agent re-plans their actions in a receding horizon manner based on estimates of market and supplier parameters thereby creating an approximate closed-loop equilibrium policy. We demonstrate through numerical simulations that this modeling approach is computationally tractable and generates economically interpretable behaviors in a variety of settings such as demand spikes, supply shocks, and information asymmetry.

Improving success probability of innovation through multi-agent collaboration: a differential game model

ABSTRACT New product or technology innovation is always facing many uncontrollable factors, which lead to the uncertainty of the success probability of an innovation project and its developing time. Multi-agent collaboration is an effective way to improve the success probability and reduce the developing time of an innovation project. In this paper, we employ the knowledge sharing model and the success probability model to formulate a differential game for the multi-agent collaborative innovation (MACI) mechanism under uncertainty. Firstly, the equilibrium policy for each agent is obtained by applying the optimal control theory. Secondly, we observe that the derived equilibrium policy can not only effectively reduce the developing time of the innovation procedure, but also improve the success probability of the innovation project. Thirdly, we show that the optimal profit allocation contract for the collaborative mechanism can be obtained by solving a linear programming problem. Finally, this paper investigates how governments’ subsidy policies affect the equilibrium of the collaborative mechanism. Numerical experiments are conducted to test the performance of different collaborative innovation mechanisms. This proposed model and its derived managerial insights provide new theoretical guidance for establishing and implementing the multi-agent collaborative mechanism of new product or technology innovations under uncertain environments.

Empirical Policy Optimization for n-Player Markov Games.

In single-agent Markov decision processes, an agent can optimize its policy based on the interaction with the environment. In multiplayer Markov games (MGs), however, the interaction is nonstationary due to the behaviors of other players, so the agent has no fixed optimization objective. The challenge becomes finding equilibrium policies for all players. In this research, we treat the evolution of player policies as a dynamical process and propose a novel learning scheme for Nash equilibrium. The core is to evolve one's policy according to not just its current in-game performance, but an aggregation of its performance over history. We show that for a variety of MGs, players in our learning scheme will provably converge to a point that is an approximation to Nash equilibrium. Combined with neural networks, we develop an empirical policy optimization algorithm, which is implemented in a reinforcement-learning framework and runs in a distributed way, with each player optimizing its policy based on own observations. We use two numerical examples to validate the convergence property on small-scale MGs, and a pong example to show the potential on large games.

A generalized Hotelling–Downs model with asymmetric candidates

AbstractWe investigate an extension of the Hotelling–Downs model to the case where the preferences of the voters do not have to be single peaked. In the case where candidates only care about winning or losing, assuming that a voter elects each candidate symmetrically with equal probability when indifferent, previous works by Fisher and Ryan and by Laffond, Laslier, and Le Breton have shown that the equilibrium outcome of the model is unique. Uniqueness also holds in the case where candidates care about the strength of their majority. Our paper shows that furthermore the equilibrium is unique for small asymmetries in voter's behavior but not necessarily for large ones. In particular, if voters always vote for the incumbent rather than a newcomer when indifferent, the equilibrium policy outcome may fail to be unique. Moreover, we provide a further sufficient condition for uniqueness. Namely, suppose all voters vote for a given candidate with the same probability when indifferent. Then, if there is a Nash equilibrium in completely mixed strategies between the candidates, it is the only Nash equilibrium.

The rate of discount on public investments with future bias in an altruistic overlapping generations model

A typical approach to evaluating public investment projects requires an appropriate rate for discounting future benefits and costs. In many developed countries, the practical discount rate for benefit–cost analysis has been calculated using the Ramsey formula in an optimal growth model, stretched back to the pioneering work presented by Frank Ramsey in 1928. Intergenerational altruism that is widely observed in our society generates future bias. Therefore, elected governments under democracies are future-biased. In such cases, the Ramsey formula is no longer valid. This paper derives a modified Ramsey formula for an appropriate social discount rate to be applied instead of the conventional Ramsey formula. We also examine properties of equilibrium policy. The degree of future bias depends on the political power of young people relative to elderly people. Demographic effects such as population aging negatively affect public investment through decreasing future bias effects. This paper contributes to the optimal theory of public investment for calculating an appropriate social discount rate, both theoretically and practically.

When Hierarchical Federated Learning Meets Stochastic Game: Toward an Intelligent UAV Charging in Urban Prosumers

Unmanned aerial vehicles (UAVs) nowadays are developing rapidly for various applications such as UAV taxis and delivery drones. However, the limited battery energy restricts the flight distance of the UAVs. Thus, urban prosumers equipped with drone recharge stations are introduced to provide charging services for the UAVs. In this article, firstly, a day-ahead energy scheduling problem for UAV charging-enabled urban prosumers is studied, where the objective is to maximize the overall energy satisfaction of the prosumers with ensuring the quality of service (QoS) of the charged UAVs. Specifically, to deal with the considered problem, we decompose it into two stages: 1) day-ahead energy requirement data prediction stage, and 2) energy scheduling stage per prosumer. Thus, secondly, a joint method based on hierarchical federated learning (HFL) on long short-term memory (LSTM) architecture (HFL-LSTM) and stochastic game-based multi-agent double deep Q-learning (MADDQN) with community agent-independent approach is proposed. In particular, the HFL-LSTM approach is leveraged to forecast each prosumer’s energy requirement data without centralized collecting local prosumers’ data such that to protect data privacy. Then, the stochastic game is adopted to analyze the formulated problem, aiming to find the Nash equilibrium strategy. Afterward, MADDQN with a community agent-independent method is utilized to achieve the best energy scheduling strategy per prosumer. Finally, the experimental results demonstrate the superiority of the proposed joint method that can achieve the lowest mean squared error with the value of 0.0152 and the highest energy satisfaction (36388) achieved by the Nash equilibrium policy compared with the benchmarks.

Learning equilibrium mean‐variance strategy

AbstractWe study a dynamic mean‐variance portfolio optimization problem under the reinforcement learning framework, where an entropy regularizer is introduced to induce exploration. Due to the time–inconsistency involved in a mean‐variance criterion, we aim to learn an equilibrium policy. Under an incomplete market setting, we obtain a semi‐analytical, exploratory, equilibrium mean‐variance policy that turns out to follow a Gaussian distribution. We then focus on a Gaussian mean return model and propose a reinforcement learning algorithm to find the equilibrium policy. Thanks to a thoroughly designed policy iteration procedure in our algorithm, we prove the convergence of our algorithm under mild conditions, despite that dynamic programming principle and the usual policy improvement theorem failing to hold for an equilibrium policy. Numerical experiments are given to demonstrate our algorithm. The design and implementation of our reinforcement learning algorithm apply to a general market setup.

Subgame-perfect equilibrium strategies for time-inconsistent recursive stochastic control problems

We study time-inconsistent recursive stochastic control problems, i.e., for which Bellman's principle of optimality does not hold. For this class of problems classical optimal controls may fail to exist, or to be relevant in practice, and dynamic programming is not easily applicable. Therefore, the notion of optimality is defined through a game-theoretic framework by means of subgame-perfect equilibrium: we interpret our preference changes which, realistically, are inconsistent over time, as players in a game for which we want to find a Nash equilibrium. The approach followed in our work relies on the stochastic (Pontryagin) maximum principle: we adapt the classical spike variation technique to obtain a characterization of equilibrium strategies in terms of a generalized second-order Hamiltonian function defined through pairs of backward stochastic differential equations, even in the multidimensional case. The theoretical results are applied in the financial field to finite horizon investment-consumption policies with non-exponential actualization. Here the existence of non-trivial equilibrium policies is also ascertained.