Linear-quadratic Case Research Articles

Symbolic dynamics for active sets of a class of constrained nonlinear optimal control and MPC problems

Solutions to optimal control problems are usually understood to provide optimal trajectories. In this paper, we show that the optimal state-space system dynamics induce a dynamics of the active sets. More specifically, given the optimal active set at the solution obtained at the current time, its successor optimal active set (which, in turn, defines the successor solution) can be found with index set operations. These operations do not involve any optimal control (or other optimization or integration) problem, but they can be described with simple rules. These rules constitute the symbolic dynamics for active sets. The present paper treats a particular constrained nonlinear problem class, extending earlier results for the constrained linear-quadratic case.

Pareto-based Stackelberg differential game for stochastic systems with multi-followers

This paper is concerned about a Pareto-based Stackelberg stochastic differential game with multi-followers, in which followers are cooperative relations. First of all, necessary and sufficient conditions for the existence of Pareto-based Stackelberg equilibrium are given for the general Pareto-based Stackelberg stochastic differential game problem. Next, we study the above game problem in the linear-quadratic (LQ) case and obtain a feedback form of Pareto-based Stackelberg equilibrium. The solvability of Riccati equations is guaranteed under some assumptions. Finally, the theoretical results are used to solve a national debt management problem.

Unified reinforcement Q-learning for mean field game and control problems

We present a Reinforcement Learning (RL) algorithm to solve infinite horizon asymptotic Mean Field Game (MFG) and Mean Field Control (MFC) problems. Our approach can be described as a unified two-timescale Mean Field Q-learning: The same algorithm can learn either the MFG or the MFC solution by simply tuning the ratio of two learning parameters. The algorithm is in discrete time and space where the agent not only provides an action to the environment but also a distribution of the state in order to take into account the mean field feature of the problem. Importantly, we assume that the agent cannot observe the population’s distribution and needs to estimate it in a model-free manner. The asymptotic MFG and MFC problems are also presented in continuous time and space, and compared with classical (non-asymptotic or stationary) MFG and MFC problems. They lead to explicit solutions in the linear-quadratic (LQ) case that are used as benchmarks for the results of our algorithm.

Non-autonomous Linear Quadratic Non-cooperative Differential Games with Continuous Updating

The subject of this paper is a non-autonomous linear quadratic case of a differential game model with continuous updating. This class of differential games is essentially new where it is assumed that, at each time instant, players have or use information about the game structure defined on a closed time interval with a fixed duration. During the interval information about motion equations and payoff functions of players updates. It is non-autonomy that simulates this effect of updating information. A linear quadratic case for this class of games is particularly important for practical problems arising in the engineering of human-machine interaction. Here we define the Nash equilibrium as an optimality principle and present an explicit form of Nash equilibrium for the linear quadratic case. Also, the case of dynamic updating for the linear quadratic differential game is studied and uniform convergence of Nash equilibrium strategies and corresponding trajectory for a case of continuous updating and dynamic updating is demonstrated.

On the asymptotic behavior of the value function in large time optimal control problems

We investigate the large time behavior of the value function associated to an optimal control problem in the finite dimensional case. We first establish the large time asymptotic expansion in the linear quadratic (LQ) case using the Hamiltonian structure of the Pontryagin Maximum Principle (PMP) extremal equations and some basic results of LQ theory. We then state, without proof, a general result for dissipative nonlinear systems.

On some estimation problems for nonlinear dynamic systems

Two problems of nonlinear guaranteed estimation for states of dynamical systems are considered. It is supposed that unknown measurable in $t$ disturbances are linearly included in the equation of motion and are additive in the measurement equations. These disturbances are constrained by nonlinear integral functionals, one of which is analog of functional of the generalized work. The studied problem consists in creation of the information sets according to measurement data containing the true position of the trajectory. The dynamic programming approach is used. If the first functional requires solving a nonlinear equation in partial derivatives of the first order which is not always possible, then for functional of the generalized work it is enough to find a solution of the linear Lyapunov equation of the first order that significantly simplifies the problem. Nevertheless, even in this case it is necessary to impose additional conditions on the system parameters in order for the system trajectory of the observed signal to exist. If the motion equation is linear in state variable, then many assumptions are carried out automatically. For this case the issue of mutual approximation of information sets via inclusion for different functionals is discussed. In conclusion, the most transparent linear quadratic case is considered. The statement is illustrated by examples.

Characterization of stochastic equilibrium controls by the Malliavin calculus

We derive a characterization of equilibrium controls in continuous-time, time-inconsistent control (TIC) problems using the Malliavin calculus. For this, the classical duality analysis of adjoint BSDEs is replaced with the Malliavin integration by parts. This results into a necessary and sufficient maximum principle which is applied to a linear-quadratic TIC problem, recovering previous results obtained by duality analysis in the mean-variance case, and extending them to the linear-quadratic setting. We also show that our results apply beyond the linear-quadratic case by treating the generalized Merton problem.

A necessary optimality condition in two-dimensional screening

This paper studies adverse selection problems with a one-dimensional principal instrument and a two-dimensional agent type. We provide an optimality condition that characterizes the bunching of continuum types. The approach is based on a reparameterization of the type space in terms of the endogenous optimal allocation level curves. The condition obtained is related with the optimality of two pooling types in the one-dimensional screening without the single-crossing. We illustrate the method by analyzing one example from the literature as well as a new example far from the linear-quadratic case

Mean field approach to stochastic control with partial information

In our present article, we follow our way of developing mean field type control theory in our earlier works [Bensoussanet al., Mean Field Games and Mean Field Type Control Theory.Springer, New York (2013)], by first introducing the Bellman and then master equations, the system of Hamilton-Jacobi-Bellman (HJB) and Fokker-Planck (FP) equations, and then tackling them by looking for the semi-explicit solution for the linear quadratic case, especially with an arbitrary initial distribution; such a problem, being left open for long, has not been specifically dealt with in the earlier literature, such as Bensoussan [Stochastic Control of Partially Observable Systems. Cambridge University Press, (1992)] and Nisio [Stochastic control theory: Dynamic programming principle. Springer (2014)], which only tackled the linear quadratic setting with Gaussian initial distributions. Thanks to the effective mean-field theory, we propose a solution to this long standing problem of the general non-Gaussian case. Besides, our problem considered here can be reduced to the model in Bandiniet al.[Stochastic Process. Appl.129(2019) 674–711], which is fundamentally different from our present proposed framework.

Study of deformation due to thermal shock in porous thermoelastic material with reference temperature dependent properties

The present study is focused on the numerical investigation in time domain of the deformation in thermoelastic material with one relaxation time (Lord and Shulman theory) with voids under dependence of modulus of elasticity and thermal conductivity on reference temperature. For dependency on reference temperature, we have taken two cases: (i) linear case and (ii) quadratic case. For numerical investigation, the finite element method with eight-/nine-noded isoparametric finite elements has been used. The results demonstrate that finite element method can reliably predict the deformation of the medium. Finite element method has an advantage over Laplace/Fourier transformation as the equations are solved directly in the time domain, but in Laplace/Fourier transformation, numerical inversion is quite complicated with possible errors. The comparison of these two cases has been discussed in result and discussion section for the temperature distribution, the displacement component, volume fraction field and the stresses. A comparison of Newmark method and Galerkin’s method for time domain has been given though three-dimensional graphs. An appreciable effect of thermal relaxation times t 0 is observed from the various results.

Recursive Feasibility of Continuous-Time Model Predictive Control Without Stabilising Constraints

We consider sampled-data Model Predictive Control (MPC) of nonlinear continuous-time control systems. We derive sufficient conditions to guarantee recursive feasibility and asymptotic stability without stabilising costs and/or constraints. Moreover, we present formulas to explicitly estimate the required length of the prediction horizon based on the concept of (local) cost controllability. For the linear-quadratic case, cost controllability can be inferred from standard assumptions. In addition, we extend results on the relationship between the horizon length and the distance of the initial state to the boundary of the viability kernel from the discrete-time to the continuous-time setting.

Necessary/sufficient conditions for Pareto optimality in finite horizon mean-field type stochastic differential game

This paper is concerned with necessary and sufficient conditions for the existence of Pareto solutions in finite horizon mean-field type stochastic cooperative differential game. Based on the equivalent characterization of Pareto optimality, the problem is transformed into a set of constrained mean-field type stochastic optimal control problems with a special structure. Utilizing the mean-field type stochastic minimum principle, the necessary conditions are put forward. Under certain convex assumptions, it is shown that the necessary conditions are also sufficient ones. Next, the indefinite linear quadratic (LQ) case is studied. It is pointed out that the solvability of two related generalized differential Riccati equations (GDREs) provides a sufficient condition under which Pareto efficient strategies are equivalent to weighted sum optimal controls. In addition, all Pareto solutions are obtained based on the solutions of two generalized differential Lyapunov equations (GDLEs). At last, an example sheds light on the effectiveness of the theoretical results.

Pareto optimality in the infinite horizon cooperative difference game

This study is concerned with the necessary and sufficient conditions for the existence of Pareto solutions in the infinite horizon cooperative difference game. Based on the assumption about the Lagrange multipliers, utilising the equivalent characterisation of the Pareto optimality, the necessary conditions for the existence of the Pareto solutions are put forward. Furthermore, two conditions are presented to guarantee that zero does not belong to the Lagrange multiplier set. In addition, it is shown that the necessary conditions are also sufficient under certain convexity assumptions and a transversality condition. Next, the indefinite linear quadratic case is discussed. For a fixed initial state, under the condition of controllability, the necessary conditions are put forward. In addition, the necessary conditions, the convexity condition on the weighted sum cost functional as well as a transversality condition provide the sufficient conditions for a control to be Pareto optimal. For an arbitrary initial state, if the system is stabilisable, then the solvability of the related algebraic Riccati equation provides a sufficient condition under which all Pareto optimal strategies are obtained by the weighted sum minimisation method.

Backward stochastic optimal control with mixed deterministic controller and random controller and its applications in linear-quadratic control

This paper investigates an optimal control problem driven by backward stochastic differential equation (BSDE) with two controllers – one is called deterministic controller and the other one is called random controller. Necessary and sufficient conditions for the mixed optimal control problem are derived. A linear-quadratic (LQ) case of the mixed optimal control problem is also studied. The mixed optimal controllers are explicitly expressed by the solution of a fully coupled mean-field forward–backward stochastic differential equation (SDE). One of novel features is that a kind of mean-field BSDE naturally arises from the research on the mixed optimal control problem. Finally, a product management problem is used to illustrate the theoretical results.

Mean Field Games With Parametrized Followers

We consider mean field games between a dominant leader and many followers, such that each follower is subject to a heterogeneous delay effect from the leader's action, who in turn can exercise governance on the population through this influence. The delay effects are assumed to be discretely distributed among the followers. Given regular enough coefficients, we describe a necessary condition for the existence of a solution for the equilibrium by a system of coupled forward–backward stochastic differential equations and stochastic partial differential equations. We provide a thorough study for the particular linear quadratic case. By adopting a functional approach, we obtain the time-independent sufficient condition, which warrants the unique existence of the solution of the whole mean field game problem. Several numerical illustrations with different time horizons and populations are demonstrated.

Geographic environmental Kuznets curves: the optimal growth linear-quadratic case

We solve a linear-quadratic model of a spatio-temporal economy using a polluting one-input technology. Space is continuous and heterogenous: locations differ in productivity, nature self-cleaning technology and environmental awareness. The unique link between locations is transboundary pollution which is modelled as a PDE diffusion equation. The spatio-temporal functional is quadratic in local consumption and linear in pollution. Using a dynamic programming method adapted to our infinite dimensional setting, we solve the associated optimal control problem in closed-form and identify the asymptotic (optimal) spatial distribution of pollution. We show that optimal emissions will decrease at given location if and only if local productivity is larger than a threshold which depends both on the local pollution absorption capacity and environmental awareness. Furthermore, we numerically explore the relationship between the spatial optimal distributions of production and (asymptotic) pollution in order to uncover possible (geographic) environmental Kuznets curve cases.

Piecewise affine parameterized value-function based bilevel non-cooperative games

Generalizing certain network interdiction games communicated to us by Andrew Liu and his collaborators, this paper studies a bilevel, non-cooperative game wherein the objective function of each player’s optimization problem contains a value function of a second-level linear program parameterized by the first-level variables in a non-convex manner. In the applied network interdiction games, this parameterization is through a piecewise linear function that upper bounds the second-level decision variable. In order to give a unified treatment to the overall two-level game where the second-level problems may be minimization or maximization, we formulate it as a one-level game of a particular kind. Namely, each player’s objective function is the sum of a first-level objective function ± a value function of a second-level maximization problem whose objective function involves a difference-of-convex (dc), specifically piecewise affine, parameterization by the first-level variables. This non-convex parameterization is a major difference from the family of games with min-max objectives discussed in Facchinei et al. (Comput Optim Appl 59(1):85–112, 2014) wherein the convexity of the overall games is preserved. In contrast, the piecewise affine (dc) parameterization of the second-level objective functions to be maximized renders the players’ combined first-level objective functions non-convex and non-differentiable. We investigate the existence of a first-order stationary solution of such a game, which we call a quasi-Nash equilibrium, and study the computation of such a solution in the linear-quadratic case by Lemke’s method using a linear complementarity formulation.

Necessary and sufficient conditions for Pareto optimality of the stochastic systems in finite horizon

This paper is concerned with the necessary and sufficient conditions for the Pareto optimality in the finite horizon stochastic cooperative differential game. Based on the necessary and sufficient characterization of the Pareto optimality, the problem is transformed into a set of constrained stochastic optimal control problems with a special structure. Utilizing the stochastic Pontryagin minimum principle, the necessary conditions for the existence of the Pareto solutions are put forward. Under certain convex assumptions, it is shown that the necessary conditions are also sufficient ones. Next, we study the indefinite linear quadratic (LQ) case. It is pointed out that the solvability of the related generalized differential Riccati equation (GDRE) provides the sufficient condition under which all Pareto efficient strategies can be obtained by the weighted sum optimality method. Two examples shed light on the effectiveness of theoretical results.

Linear-quadratic optimal control under non-Markovian switching

ABSTRACTWe study a finite-dimensional continuous-time optimal control problem on finite horizon for a controlled diffusion driven by Brownian motion, in the linear-quadratic case. We admit stochastic coefficients, possibly depending on an underlying independent marked point process, so that our model is general enough to include controlled switching systems where the switching mechanism is not required to be Markovian. The problem is solved by means of a Riccati equation, which turned out to be a backward stochastic differential equation driven by the Brownian motion and by the random measure associated with the marked point process.

A dynamic game with feedback strategies for internalizing externalities

In this paper we consider a dynamic nonzero-sum game between the polluting firms and the authorities. Although the proposed game is not easily solvable for the feedback case, i.e., it is not the linear quadratic case of game neither the degenerated game case, we calculate explicitly a stationary feedback equilibrium. In the proposed game the regulator has the ability to turn the optimal allocation of their efforts between pollution abatement and taxation of the polluting firms. During the game, the regulator’s criterion is the minimization of the total discounted costs, while the criterion of the polluting firms is their utility maximization. Next, sensitivity analyses regarding the efficiency parameters of both players are provided. The conclusions are that a farsighted regulator should put much effort in abatement measures (instead of taxation measures) as well as in the improvement of abatement efficiency.