Stochastic Control Problem Research Articles

Dynamic Optimization with Timing Risk

Timing risk refers to a situation in which the timing of an economically important event is unknown (risky) from the perspective of an economic decision maker. While this special class of dynamic stochastic control problems has many applications in economics, the methods used to solve them are not easily accessible within a single, comprehensive survey. We provide a survey of dynamic optimization methods under comprehensive assumptions about the nature of timing risk. We also relax the assumption of full information and summarize optimization with limited information, ambiguity, imperfect hedging, and dynamic inconsistency. Our goal is to provide a concise user guide for specialists and nonspecialists alike.

Dynamic margin optimization

In response to the Global Financial Crisis of 2007–2009, by now, most of the financial transactions must be cleared through central counterparties operating a dynamic margin setting mechanism. High margin calls can reduce counterparty risk in a turbulent market, but at the same time, increase liquidity risk and escalate systemic risk. In this paper, we construct a theoretical model to address this challenge, deriving an optimal margin setting policy framed as a stochastic control problem. Our analysis reveals that an adaptive, countercyclical approach is superior to a purely risk-sensitive strategy, primarily by minimizing the expected loss for the clearing institution.

A general time-inconsistent stochastic optimal control problem with delay

A general time-inconsistent stochastic optimal control problem with delay

System transformation and model-free value iteration algorithms for continuous-time linear quadratic stochastic optimal control problems

In this paper, we investigate a continuous-time linear quadratic stochastic optimal control (LQSOC) problem in an infinite horizon, where diffusion and drift terms of the corresponding stochastic system depend on both state and control variables. In light of the stochastic control theory, this LQSOC problem is reduced to solving a generalised algebraic Riccati equation (GARE). With the help of an existing model-based value iteration (VI) algorithm, we propose two data-driven VI algorithms to solve the GARE. The first one relies on transforming the stochastic system into a deterministic control system first and then solving the LQSOC problem by the data of the deterministic system. Consequently, this algorithm does not need the information of two system coefficients and has a lower algorithm complexity. The second algorithm directly uses the data generated by the stochastic system, and thus it circumvents the requirement of all system coefficients. We also provide convergence proofs of these two data-driven algorithms and validate these algorithms through two simulation examples.

Portfolio and reinsurance optimization under unknown market price of risk

We investigate the optimal investment-and-reinsurance problem for insurance company with partial information on the market price of the risk. Through the use of filtering techniques, we convert the original optimization problem involving different filtrations into an equivalent stochastic control problem under the observation filtration, i.e. the so-called separated problem. The Markovian structure of the separated problem allows us to apply a classical approach to stochastic optimization based on the Hamilton–Jacobi–Bellman equation, and to provide explicit formulas for the value function and the optimal investment-and-reinsurance strategy. We finally discuss some comparisons between the optimal strategies pursued by a partially informed insurer and that followed by a fully informed insurer, and we evaluate the value of information using the idea of indifference pricing. These results are also supported by numerical experiments.

Low rank approximation method for perturbed linear systems with applications to elliptic type stochastic PDEs

In this paper, we propose a low rank approximation method for efficiently solving stochastic partial differential equations. Specifically, our method utilizes a novel low rank approximation of the stiffness matrices, which can significantly reduce the computational load and storage requirements associated with matrix inversion without losing accuracy. To demonstrate the versatility and applicability of our method, we apply it to address two crucial uncertainty quantification problems: stochastic elliptic equations and optimal control problems governed by stochastic elliptic PDE constraints. Based on varying dimension reduction ratios, our algorithm exhibits the capability to yield a high-precision numerical solution for stochastic partial differential equations, or provides a rough representation of the exact solutions as a pre-processing phase. Meanwhile, our algorithm for solving stochastic optimal control problems allows a diverse range of gradient-based unconstrained optimization methods, rendering it particularly appealing for computationally intensive large-scale problems. Numerical experiments are conducted and the results provide strong validation of the feasibility and effectiveness of our algorithm.

Principal-Multiagents problem under equivalent changes of measure: General study and an existence result

We study a general contracting problem between the principal and a finite set of competitive agents, who perform equivalent changes of measure by controlling the drift of the output process and the compensator of its associated jump measure. In this setting, we generalize the dynamic programming approach developed by Cvitanić et al. (2018) and we also relax their assumptions. We prove that the problem of the principal can be reformulated as a standard stochastic control problem in which she controls the continuation utility (or certainty equivalent) processes of the agents. Our assumptions and conditions on the admissible contracts are minimal to make our approach work. We review part of the literature and give examples on how they are usually satisfied. We also present a smoothness result for the value function of a risk–neutral principal when the agents have exponential utility functions. This leads, under some additional assumptions, to the existence of an optimal contract.

Stochastic optimal control problem of consumption and pension insurance with uncertain lifetime and its application to real-life data

We consider a continuous-time model of optimal consumption and pension insurance for a consumer with an uncertain lifetime. In the model, the consumer earns a stochastic wage income during her working life and optimally allocates her income between personal consumption, pension insurance, and securities with a deterministic dynamic return. Due to the weak development of the stock market in developing countries, employees' income comes mainly from wages and interest on savings from banks, that are discussed in this paper. The consumer's utility and bequest functions are constant absolute risk aversion (CARA). By characterizing the optimality condition of the consumer's problem using the Hamilton-Jacobi-Bellman equation, we find the optimal consumption and pension insurance as a function of wealth in closed form. We consider an application of the model while estimating its key elements using real-life data on age-specific population size, labor income, and interest rates. We show that as the absolute risk aversion for consumption increases, consumption and wealth move in the opposite direction. We also present a novel finding that wealth and consumption can be negatively related across consumers with different levels of consumption risk aversion.

Connecting stochastic optimal control and reinforcement learning

In this paper the connection between stochastic optimal control and reinforcement learning is investigated. Our main motivation is to apply importance sampling to sampling rare events which can be reformulated as an optimal control problem. By using a parameterised approach the optimal control problem becomes a stochastic optimization problem which still raises some open questions regarding how to tackle the scalability to high-dimensional problems and how to deal with the intrinsic metastability of the system. To explore new methods we link the optimal control problem to reinforcement learning since both share the same underlying framework, namely a Markov Decision Process (MDP). For the optimal control problem we show how the MDP can be formulated. In addition we discuss how the stochastic optimal control problem can be interpreted in the framework of reinforcement learning. At the end of the article we present the application of two different reinforcement learning algorithms to the optimal control problem and a comparison of the advantages and disadvantages of the two algorithms.

Optimal Relaxed Control for a Decoupled G-FBSDE

In this paper we study a system of decoupled forward-backward stochastic differential equations driven by a G-Brownian motion (G-FBSDEs) with non-degenerate diffusion. Our objective is to establish the existence of a relaxed optimal control for a non-smooth stochastic optimal control problem. The latter is given in terms of a decoupled G-FBSDE. The cost functional is the solution of the backward stochastic differential equation at the initial time. The key idea to establish existence of a relaxed optimal control is to replace the original control problem by a suitably regularised problem with mollified coefficients, prove the existence of a relaxed control, and then pass to the limit.

A necessary conditions for optimal singular control of McKean-Vlasov stochastic differential equations driven by spatial parameters local martingale

In this paper, we focus on stochastic singular control problems involving McKean-Vlasov stochastic differential equations driven by a spatially parameterized continuous local martingale. The drift coefficient in these equations depends on the state of the solution process and its law. The control variable consists of two components: an absolutely continuous control and a singular one. Firstly, under Lipschitz conditions, we establish the existence and uniqueness of its strong solution. Next, we derive the necessary conditions for optimal singular control under the assumption that the control domain is convex. These optimality conditions differ from the classical ones in the sense that here the adjoint equation is a McKean-Vlasov backward stochastic differential equation driven by a continuous local martingale with spatial parameters. The proof of our result is based on the derivatives of the local martingale with respect to spatial parameters and the derivative of the drift coefficient with respect to the probability measure.

The life care annuity: enhancing product features and refining pricing methods

AbstractThe state-of-the-art proposes life care annuities, that have been recently designed as variable annuity contracts with Long-Term Care payouts and Guaranteed Lifelong Withdrawal Benefits. In this paper, we propose more general features for these insurance products and refine their pricing methods. We name our proposed product “GLWB-LTC”. In particular, as to the product features, we allow dynamic withdrawal strategies, including the surrender option. Furthermore, we consider stochastic interest rates, described by a Cox–Ingersoll–Ross process. As to the numerical methods, we solve the stochastic control problem involved by the selection of the optimal withdrawal strategy through a robust tree method, which outperforms the Monte Carlo approach. We name this method “Tree-LTC”, and we use it to estimate the fair price of the product, as some relevant parameters vary, such as, for instance, the entry age of the policyholder. Furthermore, our numerical results show how the optimal withdrawal strategy varies over time with the health status of the policyholder. Our findings stress the important advantage of flexible withdrawal strategies in relation to insurance policies offering protection from health risks. Indeed, the policyholder is given more choice about how much to save for protection from the possible disability states at future times.

Gradient Flows for Regularized Stochastic Control Problems

Gradient Flows for Regularized Stochastic Control Problems

Reducing Obizhaeva–Wang-type trade execution problems to LQ stochastic control problems

We start with a stochastic control problem where the control process is of finite variation (possibly with jumps) and acts as integrator both in the state dynamics and in the target functional. Problems of such type arise in the stream of literature on optimal trade execution pioneered by Obizhaeva and Wang (J. Financ. Mark. 16:1–32, 2013) (models with finite resilience). We consider a general framework where the price impact and the resilience are stochastic processes. Both are allowed to have diffusive components. First we continuously extend the problem from processes of finite variation to progressively measurable processes. Then we reduce the extended problem to a linear–quadratic (LQ) stochastic control problem. Using the well-developed theory on LQ problems, we describe the solution to the obtained LQ one and translate it back to the solution for the (extended) initial trade execution problem. Finally, we illustrate our results by several examples. Among other things, the examples discuss the Obizhaeva–Wang model with random (terminal and moving) targets, the necessity to extend the initial trade execution problem to a reasonably large class of progressively measurable processes (even going beyond semimartingales), and the effects of diffusive components in the price impact process and/or the resilience process.

Representation of stochastic optimal control problems with delay in the control variable

AbstractIn this manuscript we provide a representation in infinite dimension for stochastic optimal control problems with delay in the control variable. The main novelty consists in the fact that the representation can be applied also to dynamics where the delay in the control appears as a nonlinear term and in the diffusion coefficient. We then apply the representation to a LQ case where an explicit solution can be found.

Representation of solutions to quadratic 2BSDEs with unbounded terminal values

Second order backward stochastic differential equations (2BSDEs, for short) are one of useful tools in solving stochastic control problems with model uncertainty. In this paper, we prove a representation formula for quadratic 2BSDEs with an unbounded terminal value under a convex assumption on the generator. Because of the unboundedness of the terminal value, we are unable to use some fine properties of BMO martingales, which are often employed in the literature to deal with bounded solutions to quadratic backward stochastic differential equations. Instead, we utilize the θ-technique. We also prove an existence result under an additional assumption that the terminal value is of uniformly continuous.

Dynamic minimisation of the commute time for a one-dimensional diffusion

AbstractMotivated in part by a problem in simulated tempering (a form of Markov chain Monte Carlo) we seek to minimise, in a suitable sense, the time it takes a (regular) diffusion with instantaneous reflection at 0 and 1 to travel to 1 and then return to the origin (the so-called commute time from 0 to 1). Substantially extending results in a previous paper, we consider a dynamic version of this problem where the control mechanism is related to the diffusion’s drift via the corresponding scale function. We are only able to choose the drift at each point at the time of first visiting that point and the drift is constrained on a set of the form $$[0,\ell )\cup (i,1]$$ [ 0 , ℓ ) ∪ ( i , 1 ] . This leads to a type of stochastic control problem with infinite dimensional state.

Method for adapting radioacoustic sounding systems of the atmosphere

Stations of radioacoustic sounding (RAS) of the atmosphere are a promising means of obtaining information about the altitudinal distribution of meteorological parameters in the Earth's atmosphere used in the process of solving current scientific and applied tasks to ensure aircraft flights, weather forecasting, etc. However, the effectiveness of the existing radio-acoustic means is insufficient, and there are practical needs for the development of appropriate prospective structures and algorithms, which will be implemented during the construction of specific stations designed to solve actual applied tasks. The article presents the synthesis of the RAS systems algorithm adaptation performed by changing the frequency of the sounding radio signal to fulfill the Bragg condition when the emitted acoustic pulse signal moves along the sounding path from the standpoint of optimal control theory. Since in the problem of synthesizing the algorithm for controlling the frequency of a radio signal, the disturbing and leading to violations of the Bragg condition during the propagation of an acoustic packet along the sounding path, as well as the process causing measurement errors, are considered as random, this problem is considered as a problem of stochastic optimal control. It is shown that in accordance with the separation theorem, known from the theory of optimal stochastic control, the method of controlling the frequency of a sounding radio signal should include sequentially peformed operations of forming estimates of the information parameter of the scattered radio signal, optimal linear filtering of the obtained estimates, and deterministic control of the frequency of the sounding radio signal.

Mutual aid insurance with a three-state Markov chain

ABSTRACT Mutual aid insurance is a collective type of insurance where the policyholders share the potential losses or risks that they may face. In this paper, we establish a mathematical structure for mutual aid insurance through a three-state (good, bad and death) process, which is driven by an inhomogeneous Markov chain. The objective of maximizing an individual's lifetime utility is achieved by addressing a stochastic control problem that involves both mutual aid insurance and life insurance. We obtain the explicit expressions for optimal consumption, investment strategies, and life insurance premiums by employing the corresponding Hamilton-Jacobi-Bellman equation. In the end, we carry out a numerical analysis to show the significance of mutual aid insurance and demonstrate the optimal mutual aid insurance premium.

Stochastic optimal control problems with delays in the state and in the control via viscosity solutions and applications to optimal advertising and optimal investment problems

AbstractIn this manuscript we consider optimal control problems of stochastic differential equations with delays in the state and in the control. First, we prove an equivalent Markovian reformulation on Hilbert spaces of the state equation. Then, using the dynamic programming approach for infinite-dimensional systems, we prove that the value function is the unique viscosity solution of the infinite-dimensional Hamilton-Jacobi-Bellman equation. We apply these results to problems coming from economics: stochastic optimal advertising problems and stochastic optimal investment problems with time-to-build.