Markov Chain Approximation Method Research Articles

Power utility maximization problem under rough stochastic local volatility models with continuous-time Markov chain approximation

This paper studies the portfolio optimization problem based on rough stochastic local volatility models. Due to the challenge that this model is neither a semimartingale nor a Markov process, we first introduce the semimartingale approximate optimization problem and present the convergence analysis. We use the classic stochastic control method and continuous-time Markov chain approximation approach to solve the approximate optimization problem respectively. Then the optimal investment strategy and the HJB equation satisfied by the value function are obtained respectively. Furthermore, for the classic stochastic control method, we demonstrate the existence of solutions for the associated partial differential equation via the nonlinear Feynman–Kac formula. For continuous-time Markov chain approximation method, we prove the convergence of the value function. Finally, we provide a numerical analysis to illustrate the above results by using the power utility function. The numerical analysis shows that continuous-time Markov chain approximation method is effective in solving the portfolio optimization problem under rough stochastic local volatility models.

Finite-State Markov-Chain Approximations: A Hidden Markov Approach

This paper proposes a novel finite-state Markov chain approximation method for Markov processes with continuous support, providing both an optimal grid and transition probability matrix. The method can be used for multivariate processes, as well as non-stationary processes such as those with a life-cycle component. The method is based on minimizing the information loss between a Hidden Markov Model and the true data-generating process. We provide sufficient conditions under which this information loss can be made arbitrarily small if enough grid points are used. We compare our method to existing methods through the lens of an asset-pricing model, and a life-cycle consumption-savings model. We find our method leads to more parsimonious discretizations and more accurate solutions, and the discretization matters for the welfare costs of risk, the marginal propensities to consume, and the amount of wealth inequality a life-cycle model can generate.

Positivity-Preserving Numerical Method and Relaxed Control for Stochastic Susceptible-Infected-Vaccinated Epidemic Model with Markov Switching.

The stochastic susceptible-infected-vaccinated (SIV) epidemic model includes a nonlinear term, making it difficult to obtain analytical solutions. Thus, numerical approximation schemes are an important tool for predicting the dynamics of infectious diseases and establishing optimal control strategies. However, the convergence rate of the existing numerical methods [e.g., Euler-Maruyama (EM) and truncated EM scheme] is only 1/2 order of the time step . This article describes the construction of a logarithmic truncated EM scheme that achieves order-1 convergence and ensures positive numerical solutions of the stochastic SIV epidemic model. The existence of an invariant measure is proved for the stochastic SIV epidemic model with Markov switching. In addition, relaxed controls for the stochastic SIV epidemic model are investigated by using the Markov chain approximation method. It is demonstrated that the approximation schemes converge to the optimal strategy as the mesh size goes to zero. Finally, the results of numerical examples are presented to illustrate the theoretical results derived in this article.

A general approximation method for optimal stopping and random delay

AbstractThis study examines the continuous‐time optimal stopping problem with an infinite horizon under Markov processes. Existing research focuses on finding explicit solutions under certain assumptions of the reward function or underlying process; however, these assumptions may either not be fulfilled or be difficult to validate in practice. We developed a continuous‐time Markov chain (CTMC) approximation method to find the optimal solution, which applies to general reward functions and underlying Markov processes. We demonstrated that our method can be used to solve the optimal stopping problem with a random delay, in which the delay could be either an independent random variable or a function of the underlying process. We established a theoretical upper bound for the approximation error to facilitate error control. Furthermore, we designed a two‐stage scheme to implement our method efficiently. The numerical results show that the proposed method is accurate and rapid under various model specifications.

Harvesting of a Stochastic Population Under a Mixed Regular-Singular Control Formulation

This work focuses on optimal harvesting-renewing for a stochastic population. A mixed regular-singular control formulation with a state constraint and regime switching is introduced. The decision-makers either harvest or renew with finite or infinite harvesting/renewing rates. The payoff functions depend on the harvesting/renewing rates. Several properties of the value function are established. The limiting value function as the white noise intensity approaches infinity is identified. The Markov chain approximation method is used to find numerical approximation of the value function and optimal strategies.

A General Framework to Simulate Diffusions with Discontinuous Coefficients and Local Times

In this article, we propose an efficient general simulation method for diffusions that are solutions to stochastic differential equations with discontinuous coefficients and local time terms. The proposed method is based on sampling from the corresponding continuous-time Markov chain approximation. In contrast to existing time discretization schemes, the Markov chain approximation method corresponds to a spatial discretization scheme and is demonstrated to be particularly suited for simulating diffusion processes with discontinuities in their state space. We establish the theoretical convergence order and also demonstrate the accuracy and robustness of the method in numerical examples by comparing it to the known benchmarks in terms of root mean squared error, runtime, and the parameter sensitivity.

Mean–variance portfolio selection with dynamic attention behavior in a hidden Markov model

In this paper, we study closed-loop equilibrium strategies for mean–variance portfolio selection problems in a hidden Markov model with dynamic attention behavior. In addition to the investment strategy, the investor’s attention to news is introduced as a control of the accuracy of the news signal process. The objective is to find equilibrium strategies by numerically solving an extended HJB equation by using Markov chain approximation method. An iterative algorithm is constructed and its convergence is established. Numerical examples are provided to illustrate the results.

Applications of Markov chain approximation methods to optimal control problems in economics

In this paper we explore some benefits of using the finite-state Markov chain approximation (MCA) method of Kushner and Dupuis (2001) to solve continuous-time optimal control problems in economics. We first show that the implicit finite-difference scheme of Achdou et al. (2022) amounts to a limiting form of the MCA method for a certain choice of approximating chains and policy function iteration for the resulting system of equations. We then illustrate that relative to the implicit finite-difference approach, using variations of modified policy function iteration to solve income fluctuation problems both with and without discrete choices can lead to an increase in the speed of convergence of more than an order of magnitude. Finally, we provide several consistent chain constructions for stationary portfolio problems with correlated state variables, and illustrate the flexibility of the MCA approach by using it to construct and compare two simple solution methods for a general equilibrium model with financial frictions.

Optimal dividends for regulated insurers with a nonlinear penalty

This study investigates an optimal dividend problem for an insurer with random time regulation and a nonlinear penalty from the payment of dividends. The surplus process of the insurer is a jump–diffusion model and the goal of the insurer is to maximise the expected accumulated discount dividend payment until ruin occurs or regulatory time arrives. The introduction of the external random regulatory time leads that the HJB equation associated with this control problem is a time in-homogeneous parabolic variational inequality. There is no common explicit solution to this variational inequality. We derived explicit solutions for optimal policies when both the regulatory time and the claims follow an exponential distribution. For general cases, this paper provides a Markov chain approximation method for the value function and optimal policies and presents an example to illustrate the algorithm when the random regulatory time follows the Gompertz–Makeham distribution.

Approximating the model of a water distribution network as a Markov decision process

In this paper, our objective is to convert the model of a water distribution network described via Stochastic differential equations (SDE) into a Markov decision process (MDP). The motivation behind this work is that while MDP's represents the underlying dynamics for dynamic programming and reinforcement learning, the actual underlying model is best described via differential equations, and therefore, we would like to convert the SDE to MDP. We have applied Kushner's Markov chain approximation (MCA) method and verified it using a novel modified Monte Carlo method which can be considered as an alternative to the well-known Kushner's MCA. Both the methods approximate the value function and simulation studies show that the obtained value functions from both the methods converge to almost the same value.

A survey of numerical solutions for stochastic control problems: Some recent progress

<p style='text-indent:20px;'>This paper presents a survey on some of the recent progress on numerical solutions for controlled switching diffusions. We begin by recalling the basics of switching diffusions and controlled switching diffusions. We then present regular controls and singular controls. The main objective of this paper is to provide a survey on some recent advances on Markov chain approximation methods for solving stochastic control problems numerically. A number of applications in insurance, mathematical biology, epidemiology, and economics are presented. Several numerical examples are provided for demonstration.</p>

Convergence analysis for continuous-time Markov chain approximation of stochastic local volatility models: Option pricing and Greeks

This paper establishes the precise second order convergence rates of the continuous-time Markov chain (CTMC) approximation method for pricing options under the general framework of stochastic local volatility (SLV) models. The stochastic local volatility models studied in this paper include Heston model, 4/2 model, α-Hypergeometric model, stochastic alpha beta rho (SABR) model, Heston-SABR model and quadratic SLV model. Using the stochastic flow theorem, the closed-form CTMC approximation formula for the Greeks are obtained and the second order convergence rates are proved. Numerical examples confirm the theoretical findings.

Numerical Solutions for Optimal Control of Stochastic Kolmogorov Systems

This work is concerned with controlled stochastic Kolmogorov systems. Such systems have received much attention recently owing to the wide range of applications in biology and ecology. Starting with the basic premise that the underlying system has an optimal control, this paper is devoted to designing numerical methods for approximation. Different from the existing literature on numerical methods for stochastic controls, the Kolmogorov systems take values in the first quadrant. That is, each component of the state is nonnegative. The work is designing an appropriate discrete-time controlled Markov chain to be in line with (locally consistent) the controlled diffusion. The authors demonstrate that the Kushner and Dupuis Markov chain approximation method still works. Convergence of the numerical scheme is proved under suitable conditions.

Optimal control and numerical methods for hybrid stochastic SIS models

This work focuses on optimal controls of a class of stochastic SIS epidemic models under regime switching. By assuming that a decision maker can influence the infectivity period, our aim is to minimize the expected discounted cost due to illness, medical treatment, and the adverse effect on the society. In addition, a model with the incorporation of vaccination is proposed. Numerical schemes are developed by approximating the continuous-time dynamics using Markov chain approximation methods. It is demonstrated that the approximation schemes converge to the optimal strategy as the mesh size goes to zero. Numerical examples are provided to illustrate our results.

Switching diffusion approximations for optimal power management in parallel processing systems

In this paper, we investigate optimal power management in parallel processing systems composed of one queue and several identical processing stations. Power consumption is controlled by setting some of the stations into an inactive state, where they consume less power but are unable to provide service. This way, we are faced with the conflicting objective of minimizing power consumption while maintaining a desired quality of service. A distinguishing feature here, regarding most previous literature on this subject, is that we consider systems operating a policy that may turn the reserve machines on or off with setup times and under general inter-arrival or service time distributions, subject to some conditions. When these condition fail, we also provide a model with general inter-arrival times and exponentially distributed service times. To some extent, a controlled switching diffusion obtained in this paper via heavy traffic analysis and stochastic optimal control theory are the technical underpinning of the paper. We also propose a numerical approach to the solutions of the optimal control problems based on the Markov chain approximation method. Finally, we consider some numerical experiments that illustrate the efficiency of the proposed approach.

Analysis of Markov chain approximation for Asian options and occupation-time derivatives: Greeks and convergence rates

The continuous-time Markov chain (CTMC) approximation method is a powerful tool that has recently been utilized in the valuation of derivative securities, and it has the advantage of yielding closed-form matrix expressions suitable for efficient implementation. For two types of popular path-dependent derivatives, the arithmetic Asian option and the occupation-time derivative, this paper obtains explicit closed-form matrix expressions for the Laplace transforms of their prices and the Greeks of Asian options, through the novel use of pathwise method and Malliavin calculus techniques. We for the first time establish the exact second-order convergence rates of the CTMC methods when applied to the prices and Greeks of Asian options. We propose a new set of error analysis methods for the CTMC methods applied to these path-dependent derivatives, whose payoffs depend on the average of asset prices. A detailed error and convergence analysis of the algorithms and numerical experiments substantiate the theoretical findings.

Optimal exploitation for hybrid systems of renewable resources under partial observation

This work focuses on optimal controls for hybrid systems of renewable resources in random environments. We propose a new formulation to treat the optimal exploitation with harvesting and renewing. The random environments are modeled by a Markov chain, which is hidden and can be observed only in a Gaussian white noise. We use the Wonham filter to estimate the state of the Markov chain from the observable process. Then we formulate a harvesting–renewing model under partial observation. The Markov chain approximation method is used to find a numerical approximation of the value function and optimal policies. Our work takes into account natural aspects of the resource exploitation in practice: interacting resources, switching environment, renewing and partial observation. Numerical examples are provided to demonstrate the results and explore new phenomena arising from new features in the proposed model.

Applications of Markov Chain Approximation Methods to Optimal Control Problems in Economics

Applications of Markov Chain Approximation Methods to Optimal Control Problems in Economics

A hybrid deep learning method for optimal insurance strategies: Algorithms and convergence analysis

This paper develops a hybrid deep learning approach to find optimal reinsurance, investment, and dividend strategies for an insurance company in a complex stochastic system. A jump–diffusion regime-switching model with infinite horizon subject to ruin is formulated for the surplus process. A Markov chain approximation and stochastic approximation-based iterative deep learning algorithm is developed to study this type of infinite-horizon optimal control problems. Approximations of the optimal controls are obtained by using deep neural networks. The framework of Markov chain approximation plays a key role in building iterative algorithms and finding initial values. Stochastic approximation is used to search for the optimal parameters of neural networks in a bounded region determined by the Markov chain approximation method. The convergence of the algorithm is proved and the rate of convergence is provided.

Option pricing in exponential Lévy models with transaction costs

We present an approach for pricing European call options in presence of proportional transaction costs, when the stock price follows a general exponential L\'{e}vy process. The model is a generalization of the celebrated work of Davis, Panas and Zariphopoulou (1993), where the value of the option is defined as the utility indifference price. This approach requires the solution of two stochastic singular control problems in finite horizon, satisfying the same Hamilton-Jacobi-Bellman equation, with different terminal conditions. We introduce a general formulation for these portfolio selection problems, and then we focus on the special case in which the probability of default is ignored. We solve numerically the optimization problems using the Markov chain approximation method and show results for diffusion, Merton and Variance Gamma processes. Option prices are computed for both the writer and the buyer.