Dynamic Programming Principle Research Articles

Probabilities estimating for attacks on supply chain for critical infrastructure facilities

The different types of supply chain for critical infrastructure facilities of industrial sector were analyzed. Also, the main types of attacks in supply chain were considered. The character of resource dependencies was analysed and representation of supply chain in form of hierarchical oriented graph, with division into levels, was considered. The algorithm of taking into account of attack probabilities for objects, which giveresources for functioning of some endpoint object of supply chain was developed basing on dynamic programming principles. Calculation complexity of proposed algorithm was estimated, and it confirmed its effectiveness for practical situations. For thetarget area of use the proposed approach gives better calculation complexity in compare existing solutions.

Optimal Portfolio and Retirement Decisions with Costly Job Switching Options

In this paper, we consider the utility maximization problem of an agent regarding optimal consumption-investment, job-switching strategy, and the optimal early retirement date. The agent can switch between two jobs or job categories at any time before retirement, but incurs a cost when switching to a position offering higher labor income. The agent's utility maximization involves a combination of stochastic control for consumption and investment, switching control for job-switching, and optimal stopping for early retirement decisions, making it a non-trivial and highly challenging problem. By utilizing the dynamic programming principle, we can derive the nonlinear Hamilton-Jacobi-Bellman (HJB) equation in the form of a system of variational inequalities with obstacle constraints, which arises from the agent's optimization problem. We employ guess and verify methods based on economic intuition to derive the closed-form solution of this HJB equation and demonstrate, through a verification theorem, that this solution aligns with the solution to the agent's utility maximization problem.

Optimal Consumption and Investment with Income Adjustment and Borrowing Constraints

In this paper, we address the utility maximization problem of an infinitely lived agent who has the option to increase their income. The agent can increase their income at any time, but doing so incurs a wealth cost proportional to the amount of the increase. To prevent the agent from infinitely increasing their income and borrowing against future income, we additionally consider a non-negative wealth constraint that prohibits borrowing based on future income. This utility maximization problem is a mixture of stochastic control, where the agent chooses consumption and investment, and singular control, where the agent chooses a non-decreasing income process. To solve this non-trivial and challenging problem, we derive the Hamilton–Jacobi–Bellman (HJB) equation with a gradient constraint using the dynamic programming principle (DPP). Then, using the guess-and-verify method and a linearization technique, we obtain a closed-form solution to the HJB equation and, based on this, find the optimal strategy.

The separation theorem for mean-field linear quadratic optimal control problem with partial information

Abstract We investigate the separation principle for a class of mean-field stochastic optimal control problems with partial information, where the linear stochastic dynamic system is observed through a linear noisy channel. In our model, the time inconsistency caused by the mean-field terms makes the dynamic programming principle (DPP) inapplicable, which immediately leads to difficulties in deriving the separation principle. Utilizing the optimal filtering equations in several dimensions, we can transform the stochastic optimal control problem with partial information into one with complete details. Subsequently, the Hamilton-Jacobi-Bellman (HJB) equation that can solve the optimal control is obtained with the support of the dynamic programming principle. The separation principle for solving the optimal control of the original problem is derived.

Dynamic programming principle for optimal control of uncertain random differential equations and its application to optimal portfolio selection

This study aimed to examine an uncertain stochastic optimal control problem premised on an uncertain stochastic process. The proposed approach is used to solve an optimal portfolio selection problem. This paper’s research is relevant because it outlines the procedure for solving optimal control problems in uncertain random environments. We implement Bellman’s principle of optimality method in dynamic programming to derive the principle of optimality. Then the resulting Hamilton-Jacobi-Bellman equation (the equation of optimality in uncertain stochastic optimal control) is used to solve a proposed portfolio selection problem. The results of this study show that the dynamic programming principle for optimal control of uncertain stochastic differential equations can be applied in optimal portfolio selection. Also, the study results indicate that the optimal fraction of investment is independent of wealth. The main conclusion of this study is that, in Itô-Liu financial markets, the dynamic programming principle for optimal control of uncertain stochastic differential equations can be applied in solving the optimal portfolio selection problem.

Finite Horizon Stochastic H2/H∞ Control for Continuous‐Time Systems Driven by G‐Brownian Motion

ABSTRACTIn this article, we consider a stochastic control problem under volatility ambiguity. In this framework, the state function is described by a stochastic differential equation driven by ‐Brownian motion. Different from the classical approach, the volatility ambiguity is characterized by a family of non‐dominated probability measures. control can be modeled as a “inf‐sup problem” and control can be modeled as a “sup‐sup problem.” With the help of stochastic maximum principle and dynamic programming principle, we derive the stochastic bounded real lemma and two sets of cross coupled generalized differential Riccati equations, which is associated with the solvability of control.

The relationship between maximum principle and dynamic programming principle for stochastic recursive control problem with random coefficients

The relationship between maximum principle and dynamic programming principle for stochastic recursive control problem with random coefficients

Stochastic representation for solutions of a system of coupled HJB-Isaacs equations with integral–differential operators

In this paper, we focus on the stochastic representation of a system of coupled Hamilton–Jacobi–Bellman–Isaacs (HJB–Isaacs (HJBI), for short) equations which is in fact a system of coupled Isaacs’ type integral-partial differential equation. For this, we introduce an associated zero-sum stochastic differential game, where the state process is described by a classical stochastic differential equation (SDE, for short) with jumps, and the cost functional of recursive type is defined by a new type of backward stochastic differential equation (BSDE, for short) with two Poisson random measures, whose wellposedness and a prior estimate as well as the comparison theorem are investigated for the first time. One of the Poisson random measures μ appearing in the SDE and the BSDE stems from the integral term of the HJBI equations; the other random measure in BSDE is introduced to link the coupling factor of the HJBI equations. We show through an extension of the dynamic programming principle that the lower value function of this game problem is the viscosity solution of the system of our coupled HJBI equations. The uniqueness of the viscosity solution is also obtained in a space of continuous functions satisfying certain growth condition. In addition, also the upper value function of the game is shown to be the solution of the associated system of coupled Isaacs’ type of integral-partial differential equations. As a byproduct, we obtain the existence of the value for the game problem under the well-known Isaacs’ condition.

A Multilinear HJB-POD Method for the Optimal Control of PDEs on a Tree Structure

Optimal control problems driven by evolutionary partial differential equations arise in many industrial applications and their numerical solution is known to be a challenging problem. One approach to obtain an optimal feedback control is via the Dynamic Programming principle. Nevertheless, despite many theoretical results, this method has been applied only to very special cases since it suffers from the curse of dimensionality. Our goal is to mitigate this crucial obstruction developing a version of dynamic programming algorithms based on a tree structure and exploiting the compact representation of the dynamical systems based on tensors notations via a model reduction approach. Here, we want to show how this algorithm can be constructed for general nonlinear control problems and to illustrate its performances on a number of challenging numerical tests introducing novel pruning strategies that improve the efficacy of the method. Our numerical results indicate a large decrease in memory requirements, as well as computational time, for the proposed problems. Moreover, we prove the convergence of the algorithm and give some hints on its implementation.

Проблемы поиска и преследования беспилотных летательных аппаратов с использованием теоретико-игрового подхода

Currently, the application of mathematical modeling theory is widely used in various fields and, among other things, can be used to prevent illegal actions. This study examines scenarios involving a single pursuer tracking a single evader, as well as situations involving multiple pursuers pursuing multiple evaders. The authors formulate this problem as a search and pursuit problem for four-rotor unmanned aerial vehicles (UAVs) or quadcopters. The solution to the problem is based on game theory, as it provides a mathematical framework for modeling and studying strategic interactions involving multiple decision makers. The authors consider a game where the set of strategies of the runner is the set of possible combinations of speeds and directions of his movement, and the set of strategies of the pursuer is the set of all possible permutations of the elements of his speeds. The matrix of the resulting game consists of elements that are the time of capture. A mathematical problem on the optimal assignments of searching and pursuing unmanned aerial vehicles is formulated and solved. Purpose: optimizing the effectiveness of strategies for detecting and capturing four-rotor unmanned aerial vehicles (quadcopters). Methods: methods of mathematical modeling, game theory apparatus, Hungarian method of solving the problem of assignments, decision theory, the principle of dynamic programming, the Maple package for solving examples were used. Results: In order to fulfill the conditions for solving the problem of optimal assignments, it is necessary and sufficient that it is balanced. This assignment problem can be balanced by entering the required number of fictitious boats or escaping boats. After that, it is possible to formulate and solve the dual problem of optimal appointments. The resulting game can be solved by any method of solving matrix games. In this way, it is possible to determine the policy of harassment and search between drones. Practical importance: All escaped quadcopter UAVs can be chased and successfully intercepted using the developed models. Examples of the study of mathematical models using the Maple software package are given.

Life-cycle planning with CEV model and time-inconsistent preferences

In this paper, we investigate an optimization problem for a wage earner seeking to maximize expected utilities until retirement by choosing optimal consumption, investment, and life insurance purchase strategies. The constant elasticity of variance (CEV) model is adopted to describe the price process of the risky asset. Additionally, we assume that the wage earner has time-inconsistent preferences. This makes the wage earner discount her payoff by a non-constant discount rate. Applying the dynamic programming principle, we have derived the Hamilton-Jacobi-Bellman (HJB) equation corresponding to the optimization problem. Furthermore, we present semi-analytical expressions for optimal strategies and value functions in three cases: the benchmark model with time-consistent preferences, the naive and sophisticated wage earners with time-inconsistent preferences. Finally, illustrations of the optimal solutions and some economic insights are provided in the numerical examples.

Enhanced Dynamic Programming Approaches for Efficient Solutions to the Traveling Salesman Problem

This study aims to enhance dynamic programming techniques for efficiently solving the Traveling Salesman Problem, a fundamental combinatorial optimization challenge. Given its NP-hard classification, traditional exact algorithms become computationally infeasible as the problem size increases. The research revisits foundational dynamic programming principles, notably the Held-Karp algorithm, and identifies existing limitations. The study begins with a comprehensive literature review, followed by an analysis of the dynamic programming challenges specific to TSP. Novel algorithms are then developed, implemented, and rigorously tested against benchmark instances. Performance evaluation is conducted using metrics such as execution time, memory usage, and solution optimality across different problem sizes. Results demonstrate significant improvements in efficiency and scalability, with enhanced algorithms achieving optimal solutions in reduced time and computational resource usage. However, the exponential growth in complexity remains a challenge for larger instances. The study concludes with recommendations for future research, focusing on further algorithmic refinements and exploring hybrid approaches to address large-scale TSPs.

Risk-Averse Markov Decision Processes Through a Distributional Lens

By adopting a distributional viewpoint on law-invariant convex risk measures, we construct dynamic risk measures (DRMs) at the distributional level. We then apply these DRMs to investigate Markov decision processes, incorporating latent costs, random actions, and weakly continuous transition kernels. Furthermore, the proposed DRMs allow risk aversion to change dynamically. Under mild assumptions, we derive a dynamic programming principle and show the existence of an optimal policy in both finite and infinite time horizons. Moreover, we provide a sufficient condition for the optimality of deterministic actions. For illustration, we conclude the paper with examples from optimal liquidation with limit order books and autonomous driving. Funding: This work was supported by Natural Sciences and Engineering Research Council of Canada [Grants RGPAS-2018-522715 and RGPIN-2018-05705].

Modeling mispricing risk of defined contribution pension plan with a mean–variance criteria

This manuscript addresses modeling mispricing risk of defined contribution pension plan (DCPP) with a mean–variance criterion to obtain the optimal investment strategy. Provides a way for the sustainability of pensions by investing in the financial market. The pension manager’s objective is to maximize the expected terminal wealth while simultaneously minimizing the associated risk. We employ the stochastic dynamic programming principle (SDPP) and the Lagrange dual theorem to derive the efficient frontier and strategy, then two special cases are examined. Last, we conduct a numerical analysis to show how different parameters influence the efficient frontier and strategy. This analysis sheds light on the economic implications of our findings.

Approaches and Methods of Optimazing Long-Term Strategy of Pulling Projects of Nuclear Legacy out of Operation

The article studies a new approach to shaping possible variants of long-term strategies of pulling projects of nuclear legacy out of operation, which is based on optimization with criteria on min costs and restrictions on resources, radioactive risks and duration of the implementation period. The author put forward different variants of criteria and restrictions of strategy of pulling projects of nuclear legacy out of operation. Methods of resolving the task of optimization were studied that are based on principles of dynamic programming and an example of getting a solution was demonstrated. Benefits and drawbacks of the approach were discussed in comparison with accepted in global and Russian practice methods of prioritization of the procedure of pulling projects out of operation. The article analyzed prospects of using the proposed approach for the development of long-term road map aimed at settling problems of nuclear legacy in the Russian Federation.

Investment–consumption optimization with transaction cost and learning about return predictability

In this paper, we investigate an investment–consumption optimization problem in continuous-time settings, where the expected rate of return from a risky asset is predictable with an observable factor and an unobservable factor. Based on observable information, a decision-maker learns about the unobservable factor while making investment–consumption decisions. Both factors are supposed to follow a mean-reverting process. Also, we relax the assumption for perfect liquidity of the risky asset through incorporating proportional transaction costs that are incurred in trading the risky asset. In such way, a form of friction posing liquidity risk to the investor is examined. Dynamic programming principle coupled with an Hamilton–Jacobi–Bellman (HJB) equation are adopted to discuss the problem. Applying an asymptotic method with small transaction costs being taken as a perturbation parameter, we determine the frictional value function by solving the first and second corrector equations. For the numerical implementation of the proposed approach, a Monte-Carlo-simulation-based approximation algorithm is adopted to solve the second corrector equation. Finally, numerical examples and their economic interpretations are discussed.

Duality in optimal consumption–investment problems with alternative data

This study investigates an optimal consumption–investment problem in which the unobserved stock trend is modulated by a hidden Markov chain that represents different economic regimes. In the classic approach, the hidden state is estimated using historical asset prices, but recent technological advances now enable investors to consider alternative data in their decision-making. These data, such as social media commentary, expert opinions, COVID-19 pandemic data and GPS data, come from sources other than standard market data sources but are useful for predicting stock trends. We develop a novel duality theory for this problem and consider a jump-diffusion process for alternative data series. This theory helps investors identify “useful” alternative data for dynamic decision-making by providing conditions for the filter equation that enable the use of a control approach based on the dynamic programming principle. We apply our theory to provide a unique smooth solution for an agent with constant relative risk aversion once the distributions of the signals generated from alternative data satisfy a bounded likelihood ratio condition. In doing so, we obtain an explicit consumption–investment strategy that takes advantage of different types of alternative data that have not been addressed in the literature.

Optimal control of stochastic differential equations with random impulses and the Hamilton–Jacobi–Bellman equation

AbstractIn this article, we study the optimal control of stochastic differential equations with random impulses. We optimize the performance index and add the influence of random impulses to the performance index with a random compensation function. Using the idea of stochastic analysis and dynamic programming principle, a new Hamilton–Jacobi–Bellman (HJB) equation is obtained, and the existence and uniqueness of its viscosity solution are proved.

Viscosity solutions approach to finite-horizon continuous-time Markov decision process

This paper investigates the optimal control problems for the finite-horizon continuous-time Markov decision processes with delay-dependent control policies. We develop compactification methods in decision processes and show that the existence of optimal policies. Subsequently, through the dynamic programming principle of the delay-dependent control policies, the differential-difference Hamilton-Jacobi-Bellman (HJB) equation in the setting of discrete space is established. Under certain conditions, we give the comparison principle and further prove that the value function is the unique viscosity solution to this HJB equation. Based on this, we show that among the class of delay-dependent control policies, there is an optimal one which is Markovian.

Stochastic processes under parameter uncertainty

In this paper we study a family of nonlinear (conditional) expectations that can be understood as a stochastic process with uncertain parameters. We develop a general framework which can be seen as a version of the martingale problem method of Stroock and Varadhan with parameter uncertainty. To illustrate our methodology, we explain how it can be used to model nonlinear Lévy processes in the sense of Neufeld and Nutz, and we introduce the new class of stochastic partial differential equations under parameter uncertainty. Moreover, we study properties of the nonlinear expectations. We prove the dynamic programming principle, i.e., the tower property, and we establish conditions for the (strong) USCb–Feller property and a strong Markov selection principle.