Bellman's Principle Research Articles

Enhanced trajectory tracking control algorithm for ROVs considering actuator saturation, external disturbances, and model parameter uncertainties

In this study, an augmented model-based predictive control (AMPC) algorithm was designed for a work-class remotely operated vehicles (ROV) with actuator saturation constraints, model parameter uncertainty, and external disturbances. Firstly, model parameter uncertainties and external disturbances (primarily umbilical cable forces) were considered in the kinematic model of the ROV. The ROV's actuator saturation constraints were introduced when designing the optimal control problem (OCP). Secondly, Bellman's principle of optimality was applied to the solution process of the OCP, decomposing it into multiple easily solvable sub-optimization problems to reduce computational complexity. Then, the recursive feasibility and input-to-state practical stability (ISpS) of the AMPC algorithm were proven in the presence of model parameter uncertainties and external disturbance forces. It was also demonstrated that the state convergence region of the system employing the AMPC algorithm was smaller than that of the system using the min-max MPC algorithm when reaching a steady state. Finally, numerical simulations verified that under the influence of external disturbance forces, model parameter uncertainties, and positioning system noise, the ROV using the AMPC algorithm achieved higher accuracy in tracking the desired trajectory compared to the min-max MPC algorithm.

Bellman Neural Networks for the Class of Optimal Control Problems With Integral Quadratic Cost

This work introduces Bellman Neural Networks (BeNNs) and employs them to learn the optimal control actions for the class of optimal control problems (OCPs) with integral quadratic cost. BeNNs represent a particular family of Physics-Informed Neural Networks (PINNs) specifically designed and trained to tackle OCPs via applying the Bellman Principle of Optimality (BPO). The BPO provides necessary and sufficient optimality conditions, which result in a nonlinear partial differential equation known as the Hamilton-Jacobi-Bellman (HJB) equation. BeNNs learn the optimal control actions from the unknown solution of the arising HJB equation (i.e., the value function), where the unknown solution is modeled using a Neural Network. Additionally, the paper shows how to estimate the upper bounds on the generalization error of BeNNs while learning the solutions for the OCP class under consideration. The generalization error estimate is provided in terms of the choice and number of the training points as well as the training error. Numerical studies show that BeNNs can be successfully applied to learn the feedback control actions for the class of optimal control problems considered and, after the training is completed, deployed to control the system in a closed-loop fashion. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Impact Statement</i> —The proposed research improves our understanding of how to solve optimal control problems with closed-loop solutions and has potentially a countless number of applications in several different areas. The study is at the intersection between optimal control theory and artificial intelligence connected with mathematical tools for functional interpolation. This advances the ability to implement a higher level of autonomy in decision-making for practical applications with a beneficial impact on our society.

Исследование предиктивных схем управления функционированием организационно-технических систем

Organizational and technical systems are the most common tool for solving national economic problems. However, due to the personnel included in them, they are characterized by stochastic behavior, which raises the issue of developing their probabilistic model. On the other hand, the probabilistic apparatus is a tool for forecasting future phenomena and situations and therefore a mechanism for its use in predictive (forecast) management is needed. For this purpose, the work uses the first probabilistic approximation in the form of a Markov process and its description by the Fokker–Planck–Kolmogorov equation. The purpose of the study is to formulate and solve optimal control problems with probabilistic control quality criteria for a tracking predictive control scheme, as well as to develop a mechanism for metasystem switching of mo¬dels in the case of an adaptive scheme with a predictive model in the control loop. Materials and methods. Based on the analysis of scientific ideas and methodological approaches in predictive management of domestic and foreign authors, as well as mathematical methods and models, two schemes for implementing predictive management were selected to improve the efficiency of the functioning of organizational and technical systems: tracking schemes and adaptive management. Results. The optimal control problem was solved using two methods: Pontryagin's maximum principle and Bellman's principle of dynamic programming. It is shown that in the first case there is a convex upward strategy for achieving the target value of the controlled value with a maximizing approach, and in the second case the result is a convex downward strategy used for the optimal consumption of management resources. In addition, for another predictive control scheme – adaptive with a predictive model in the loop, the effectiveness of using the strategy of metasystem switching of models with the formulation and solution of six metasystem problems has been proven. Conclusion. The proposed approach can be used when designing management systems in organizational and technical systems and choosing behavior strategies in difficult market conditions when solving problems of their development.

Transition density function expansion methods for portfolio optimization

AbstractIn this study, we introduce transition density function expansion methods inspired from Yang et al. (J Econom. 2019;209(2):256–288.) to stochastic control issues related to utility maximization, without imposing limitations on the variety of asset price models and utility functions. Utilizing Bellman's dynamic programming principle, we initially recast the conditional expectation via the transition density function pertinent to the diffusion process. Subsequently, we employ the Itô‐Taylor expansion and Delta expansion techniques to the transition density function associated with the multivariate diffusion process, facilitated by a quasi‐Lamperti transformation, aiming to derive explicit recursive expressions for expansion coefficient functions. Our main contributions are that we articulate detailed algorithms, stemming from the backward recursive formulations of the value function and optimal strategies, achieved through discretization methodologies with rigorous proof of expansion convergence in portfolio optimization. Both theoretical and practical demonstrations are presented to validate the convergence of these approximate techniques in addressing stochastic control challenges. To underscore the efficiency and precision of our proposed methods, we apply them to portfolio selection problems within several benchmark models, and highlight the reduced complexity in comparison to the current methodologies.

On principle of optimality for safety-constrained Markov Decision Process and p-Safe Reinforcement Learning ⋆

We study optimality for the safety-constrained Markov decision process which is the underlying framework for safe reinforcement learning. Specifically, we consider an undiscounted safety-constrained Markov decision process subject to random stopping times. The decision maker's goal is to reach a goal state while avoiding unsafe states with certain probabilistic guarantees. Therefore the underlying Markov chain for any control policy will be Multichain or non-ergodic since by definition there exists a goal set and an unsafe set. Bellman's principle of optimality does not hold for such a safety-constrained Markov decision process in a Multichain setting as highlighted by a counterexample. We resolve the aforementioned counterexample by considering a zero-sum game setting between the policy and the Lagrange multiplier vector. Under suitable assumptions regarding the existence of admissible policy, we propose an off-policy RL algorithm for learning an optimal policy that satisfies the probabilistic safety guarantees. After that, we present the finite time error bound of the proposed RL algorithm. Lastly, we present simulation results of the aforementioned RL algorithm on a robot in a grid world setting.

Modified dynamic programming for asteroids belt exploration

In the past, space trajectory optimisation was limited to the design of space transfers to single destinations. More recent mission concepts however present the challenge to target multiple destinations; be it for science, exploration or even exploitation. This paper focuses on space trajectories that aim to visit multiple main-belt asteroids with one single spacecraft. The trajectory design of these missions is complicated by the fact that the asteroid sequence is not known a priori, but it is the objective of the optimisation problem. Usually, these problems are tackled as global optimisation problems under the formulation of Mixed-Integer Non-Linear Programming, on which the design variables assume both continuous and discrete values. However, beyond the aim of finding the global optimum, mission designers are usually interested in providing a wide range of mission design options reflecting the multi-modality of the problem. The paper describes a multi-fidelity design pipeline that enables a consistent exploration of all feasible asteroid tours with a given set of mission boundary conditions (e.g., launch window, mission duration, Δv). The consistency of the exploration is ensured by a deterministic search scheme which retains the n-best paths at each stage of a dynamic programming process. Bellman's principle of optimality is made applicable by transcribing the search space into a series of discretized events defined by asteroid index and fly-by epoch, as well as ensuring optimal substructure property with the choice of transfer model and design variables. The method is deployed to explore asteroid tour opportunities visiting 10 or more main belt objects and mission boundary conditions compatible with medium or discovery class missions. 79 million such trajectories are found with launch date in 2037.

Deterministic differential games in infinite horizon involving continuous and impulse controls

ABSTRACT We study a new class of two-player, zero-sum, deterministic differential games where each player uses both continuous and impulse controls in an infinite horizon with discounted payoff. We assume that the form and cost of impulses depend on nonlinear functions and the state of the system, respectively. We use Bellman's dynamic programming principle (DPP) and viscosity solutions approach to show, for this class of games, the existence and uniqueness of a solution for the associated Hamilton–Jacobi–Bellman–Isaacs (HJBI) partial differential equations (PDEs). We then, under Isaacs' condition, deduce that the lower and upper value functions coincide, and we give a computational procedure with a numerical test for the game.

Subgame-perfect equilibrium strategies for time-inconsistent recursive stochastic control problems

We study time-inconsistent recursive stochastic control problems, i.e., for which Bellman's principle of optimality does not hold. For this class of problems classical optimal controls may fail to exist, or to be relevant in practice, and dynamic programming is not easily applicable. Therefore, the notion of optimality is defined through a game-theoretic framework by means of subgame-perfect equilibrium: we interpret our preference changes which, realistically, are inconsistent over time, as players in a game for which we want to find a Nash equilibrium. The approach followed in our work relies on the stochastic (Pontryagin) maximum principle: we adapt the classical spike variation technique to obtain a characterization of equilibrium strategies in terms of a generalized second-order Hamiltonian function defined through pairs of backward stochastic differential equations, even in the multidimensional case. The theoretical results are applied in the financial field to finite horizon investment-consumption policies with non-exponential actualization. Here the existence of non-trivial equilibrium policies is also ascertained.

A Minimum Principle for Stochastic Optimal Control Problem with Interval Cost Function

In this paper, we study an optimal control problem in which their cost function is interval-valued. Also, a stochastic differential equation governs their state space. Moreover, we introduce a generalized version of Bellman's optimality principle for the stochastic system with an interval-valued cost function. Also, we obtain the Hamilton–Jacobi–Bellman equations and their control decisions. Two numerical examples happen in finance in which their cost function are interval-valued functions, illustrating the efficiency of the discussed results. The obtained results provide significantly reliable decisions compared to the case where the conventional cost function is applied.

Relationships between the maximum principle and dynamic programming for infinite dimensional stochastic control systems

The Pontryagin type maximum principle and Bellman's dynamic programming principle serve as two of the most important tools in solving optimal control problems. There is a huge literature on the study of relationship between them. The main purpose of this paper is to investigate the relationship between the Pontryagin type maximum principle and the dynamic programming principle for control systems governed by stochastic evolution equations in infinite dimensional space, with the control variables entering into both the drift and the diffusion terms. To do so, we first prove the dynamic programming principle for those systems without employing the martingale solutions. Then we establish the desired relationships in both cases that value function associated is smooth or nonsmooth. For the nonsmooth case, in particular, by employing the relaxed transposition solution, we discover the connection between the superdifferentials and subdifferentials of value function and the first-order and second-order adjoint equations.

Uncertain random problem for multistage switched systems

&lt;abstract&gt;&lt;p&gt;Optimal control problems for switched systems how best to switch between different subsystems. In this paper, two kinds of linear quadratic optimal control problems for multistage switched systems composing of both randomness and uncertainty are studied. Chance theory brings us a useful tool to deal with this indeterminacy. Based on chance theory and Bellman's principle, the analytical expressions are derived for calculating both the optimal control input and the optimal switching control law. Optimal control is implemented by genetic algorithm instead of enumerating all the elements of a series of sets whose size grows exponentially. Finally, the results of numerical examples are provided to illustrate the effectiveness of the proposed method.&lt;/p&gt;&lt;/abstract&gt;

Integer optimal control problems with total variation regularization: Optimality conditions and fast solution of subproblems

We investigate local optimality conditions of first and second order for integer optimal control problems with total variation regularization via a finite-dimensional switching-point problem. We show the equivalence of local optimality for both problems, which will be used to derive conditions concerning the switching points of the control function. A non-local optimality condition treating back-and-forth switches will be formulated. For the numerical solution, we propose a proximal-gradient method. The emerging discretized subproblems will be solved by employing Bellman’s optimality principle, leading to an algorithm which is polynomial in the mesh size and in the admissible control levels. An adaption of this algorithm can be used to handle subproblems of the trust-region method proposed in Leyffer and Manns, ESAIM: Control Optim. Calc. Var. 28 (2022) 66. Finally, we demonstrate computational results.

МАТЕМАТИЧЕСКАЯ МОДЕЛЬ ЗАДАЧИ О ЗАМЕНЕ ОБОРУДОВАНИЯ

The study was carried out with the aim of solving the problem of replacing equipment and creating a mathematical model of the problem of controlled optimization, as the most complete and realistic description of the production process. The Bellman principle of optimality was applied in the study, since when solving dynamic programming problems, this principle is general. For the solution, the Bellman functional equation was chosen as the initial mathemat cal model: &#x0D; &#x0D; &#x0D; &#x0D; &#x0D; &#x0D; The advantage of such a mathematical model is the possibility of modifying the problem. At the first step, it is proposed to sell the replaced equipment at the price (R(tk)-Z(tk))p, 0&lt;p&lt;1, p is the markdown coefficient for the sale. Further, the installed equipment may not be new and purchased at a price ((R(tk)-Z(tk))q, 0&lt;q&lt;1. Here q is the discount factor for purchase and is set at the input. Not new equipment is cheaper than new and sale of replaced equipment also contributes to profits New components must be added to the control vector - the age of the equipment being installed (not new) The control structure becomes more complex and realistic The effect of the "curse of dimensionality" does not appear during numerous tests of the program Three modifications of the problem have been developed on equipment replacement. Replaced equipment is thrown away. You can not throw it away, but sell it at a price (Rk(tk)-Zk(tk))p, where p is a markdown coefficient 0&lt;p&lt;1. You can increase profits if you buy not new equipment The paper considers the option of adding another component to the control vector - "rejuvenating" repair of equipment. After such repair, the age of the equipment becomes, for example, one year less. Repairs are taken from the graph of profit earned versus repair cost, which is obtained from the results of the program runs. Combining the above modifications into a single whole is a mathematical model of the problem of replacing equipment, which is the basis of the program.

The Hopf-Lax formula for multiobjective costs with non-constant discount via set optimization

The minimization of a multiobjective Lagrangian with non-constant discount is studied. The problem is embedded into a set-valued framework and a corresponding definition of the value function is given. Bellman's optimality principle and Hopf-Lax formula are derived. The value function is shown to be a solution of a set-valued Hamilton-Jacobi equation.

Recurrent Model Predictive Control: Learning an Explicit Recurrent Controller for Nonlinear Systems

This paper proposes an offline control algorithm, called Recurrent Model Predictive Control (RMPC), to solve large-scale nonlinear finite-horizon optimal control problems. It can be regarded as an explicit solver of traditional Model Predictive Control (MPC) algorithms, which can adaptively select appropriate model prediction horizon according to current computing resources, so as to improve the policy performance. Our algorithm employs a recurrent function to approximate the optimal policy, which maps the system states and reference values directly to the control inputs. The output of the learned policy network after N recurrent cycles corresponds to the nearly optimal solution of N-step MPC. A policy optimization objective is designed by decomposing the MPC cost function according to the Bellman's principle of optimality. The optimal recurrent policy can be obtained by directly minimizing the designed objective function, which is applicable for general nonlinear and non input-affine systems. Both simulation-based and real-robot path-tracking tasks are utilized to demonstrate the effectiveness of the proposed method.

Expected Value Model of an Uncertain Production Inventory Problem with Deteriorating Items

In this study, we present an optimal control model for an uncertain production inventory problem with deteriorating items. The dynamics of the model includes perturbation by an uncertain canonical process. An expected value optimal control model is established based on the uncertainty theory. The aim of this study is to apply the optimal control theory to solve a production inventory problem with deteriorating items and derive an optimal inventory level and production rate that would maximize the expected revenue. The uncertainty theory is used to obtain the equation of optimality. The Hamilton–Jacobi–Bellman (HJB) principle is used to solve the equation of optimality. The results are discussed using numerical experiments for different demand functions.

Accurate Current Sharing and Voltage Regulation in Hybrid Wind/Solar Systems: An Adaptive Dynamic Programming Approach

Renewable energy is an advisable choice to reduce fuel consumption and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\rm CO_{2}$ </tex-math></inline-formula> emission. Therein, wind energy and solar energy are the most promising contributors to reach this goal. Although the hybrid wind/solar system has been widely studied, the real-time current sharing based on their maximum capacities is rarely achieved in terms of seconds. Based on this, this paper proposes an accurate current sharing and voltage regulation approach in hybrid wind/solar systems, which is based on distributed adaptive dynamic programming approach. Firstly, the equivalent wind/solar model is built, which is an indispensable preprocessing to achieve the complementary between wind energy and solar energy. Therein, the wind energy and solar energy can output relative current according to their respective capacity ratio, which ensure the maximum utilization ratio of renewable energy source. Furthermore, current sharing and voltage regulation problem is switched into optimal control problem. Under this effect, each source agent aims to obtain the optimal control variable and achieve accurate current sharing/voltage regulation. Moreover, an adaptive dynamic programming approach based on Bellman principle is proposed. It can achieve accurate current sharing and voltage regulation. Finally, the simulation results are provided to illustrate the performance of the proposed adaptive dynamic programming approach.

Acceptability maximization

&lt;p style='text-indent:20px;'&gt;The aim of this paper is to study the optimal investment problem by using coherent acceptability indices (CAIs) as a tool to measure the portfolio performance. We call this problem the acceptability maximization. First, we study the one-period (static) case, and propose a numerical algorithm that approximates the original problem by a sequence of risk minimization problems. The results are applied to several important CAIs, such as the gain-to-loss ratio, the risk-adjusted return on capital and the tail-value-at-risk based CAI. In the second part of the paper we investigate the acceptability maximization in a discrete time dynamic setup. Using robust representations of CAIs in terms of a family of dynamic coherent risk measures (DCRMs), we establish an intriguing dichotomy: if the corresponding family of DCRMs is recursive (i.e. strongly time consistent) and assuming some recursive structure of the market model, then the acceptability maximization problem reduces to just a one period problem and the maximal acceptability is constant across all states and times. On the other hand, if the family of DCRMs is not recursive, which is often the case, then the acceptability maximization problem ordinarily is a time-inconsistent stochastic control problem, similar to the classical mean-variance criteria. To overcome this form of time-inconsistency, we adapt to our setup the set-valued Bellman's principle recently proposed in [&lt;xref ref-type="bibr" rid="b23"&gt;23&lt;/xref&gt;] applied to two particular dynamic CAIs - the dynamic risk-adjusted return on capital and the dynamic gain-to-loss ratio. The obtained theoretical results are illustrated via numerical examples that include, in particular, the computation of the intermediate mean-risk efficient frontiers.&lt;/p&gt;

Structural-parametric synthesis of the model of multi-criteria optimization of the solution of training problems with an aviation multifunctional simulator

The basis for the training of aircraft crews is the solution of a finite set of training tasks modeled using aviation multifunction simulators (AMT). To develop a model of multi-criteria choice of solutions for an individual and the entire set of educational tasks, its structural and parametric synthesis is carried out on the basis of the optimal distribution of AMT hardware and software resource for modeling various factors and conditions of information processes during educational tasks. Their solution is based on the search for optimal options on the set of branching and / or repeating information flows in the form of a set of elementary operations on the set of performance criteria, taking into account the principles and preferences of instructors - decision makers. The choice of a solution is carried out by summarizing single-criteria schemes for solving a particular educational problem using the Bellman principle and building on their basis a model of multi-criteria selection (MVR). The basis of the generalized MVR is the representation of AMT in the form of a directed graph in which the vertices characterize the learning tasks, and the arcs determine the possible solutions to them in the form of a certain set of elementary operations with optimal weight vectors connecting the initial and final vertices. The essence of the model is the sequential combination of the paths of elementary operations taking into account the integrity of the presentation of the educational task and the narrowing of their number through the selection function implemented using the dominance and blocking algorithms in the binary relation, which has the property of transitivity and asymmetry. The choice of a generalized solution is carried out on the basis of Pareto optimal solutions, constructed by extrapolation of expert estimates of decision-makers on a cone of preferences for various situations related to computer computing capabilities. The generalized solution of the MVR substantiation problem is based on the selection of optimal routing routes for elementary operations on the set of multidisciplinary training tasks solved by AMT.

Optimal investment strategies for pension funds with regulation-conform dynamic pension payment management in the absence of guarantees

In this article we consider the post-retirement phase optimization problem for a specific pension product in Germany that comes without guarantees. The continuous-time optimization problem is defined consisting of two specialties: first, we have a product-specific pension adjustment mechanism based on a certain capital coverage ratio which stipulates compulsory pension adjustments if the pension fund is underfunded or significantly overfunded. Second, due to the retiree’s fear of and aversion against pension reductions, we introduce a total wealth distribution to an investment portfolio and a buffer portfolio to lower the probability of future potential pension shortenings. The target functional in the optimization, that is to be maximized, is the client’s expected accumulated utility from the stochastic future pension cash flows. The optimization outcome is the optimal investment strategy in the proposed model. Due to the inherent complexity of the continuous-time framework, the discrete-time version of the optimization problem is considered and solved via the Bellman principle. In addition, for computational reasons, a policy function iteration algorithm is introduced to find a stationary solution to the problem in a computationally efficient and elegant fashion. A numerical case study on optimization and simulation completes the work with highlighting the benefits of the proposed model.