Bellman's Dynamic Programming Research Articles

Analysis of the technical condition of building structures on the example of a feed preparation workshop

The article aims to develop an instrumental model for a complex assessment of building structures that are subject to changes during their life-cycle process. This model can be used to substantiate the need either to reconstruct the existing building structures with the purpose of creating favorable living conditions or to demolish the building. The research methodology involved the general theory of systems and available techniques for estimating the state of building structures. Brick walls were tested by a nondestructive method using an IPS-MG-4.03 meter. Gypsum block masonry walls were tested by an ONIKS- 2.5 meter. Concrete was tested by an OMSh-1 hardness tester. Bellman dynamic programming was used to optimize the calculations and to find a solution for optimizing reconstruction costs. An example of assessing the condition of a building construction is provided, along with measures for extending its lifecycle. A conclusion is made that evaluation of the reliable operation of a building structure should consider the integral condition of the basic load-bearing structures under the action of environmental factors. This allows the risks of building destruction and the costs of building reconstruction to be reduced. For evaluating the current condition of the building under study, its structure was divided into a number of elements: foundation, wall box, floor slabs, roof. The hydrogeological parameters of the soil were also considered. Basing on the wear of building elements, the weakening coefficients of the building were calculated. The transient states of the system were described by semi-Markov processes. The results obtained can be used when evaluating the building residual life, selecting measures for improving the durability of structures with high wear and tear, and justifying the renovation concepts of territories.

Maximum Principle in Autonomous Multi-Object Safe Trajectory Optimization

The following article presents the task of optimizing the control of an autonomous object within a group of other passing objects using Pontryagin’s bounded maximum principle. The basis of this principle is a multidimensional nonlinear model of the control process, with state constraints reflecting the motion of passing objects. The analytical synthesis of optimal multi-object control became the basis for the algorithm for determining the optimal and safe object trajectory. Simulation tests of the algorithm on the example of real navigation situations with various numbers of objects illustrate their safe trajectories in changing environmental conditions. The optimal object trajectory obtained using Pontryagin’s maximum principle was compared with the trajectory calculated using the Bellman dynamic programming method. The analysis of the research allowed for the formulation of valuable conclusions and a plan for further research in the field of autonomous vehicle control optimization. The maximum principle algorithm allows one to take into account a larger number of objects whose data are derived from ARPA anti-collision radar systems.

Transition density function expansion methods for portfolio optimization

AbstractIn this study, we introduce transition density function expansion methods inspired from Yang et al. (J Econom. 2019;209(2):256–288.) to stochastic control issues related to utility maximization, without imposing limitations on the variety of asset price models and utility functions. Utilizing Bellman's dynamic programming principle, we initially recast the conditional expectation via the transition density function pertinent to the diffusion process. Subsequently, we employ the Itô‐Taylor expansion and Delta expansion techniques to the transition density function associated with the multivariate diffusion process, facilitated by a quasi‐Lamperti transformation, aiming to derive explicit recursive expressions for expansion coefficient functions. Our main contributions are that we articulate detailed algorithms, stemming from the backward recursive formulations of the value function and optimal strategies, achieved through discretization methodologies with rigorous proof of expansion convergence in portfolio optimization. Both theoretical and practical demonstrations are presented to validate the convergence of these approximate techniques in addressing stochastic control challenges. To underscore the efficiency and precision of our proposed methods, we apply them to portfolio selection problems within several benchmark models, and highlight the reduced complexity in comparison to the current methodologies.

Optimal investment–consumption–insurance strategy with inflation risk and stochastic income in an Itô–Lévy setting

This paper’s focus is on finding the optimal strategies for a trader who invests in stock, a money market account and an inflation-linked index bond. The stock follows a jump diffusion process and the bond is linked to inflation making the two risky. The optimal strategies are determined on two generations of the life of an investor, that is before the investor dies and after the investor dies. We applied the concept of change of probability measures considering Girsanov’s and the Radon–Nikodym theorems. We found the generator of the Backward Stochastic differential equations defined and employed the Hamilton–Jacobi–Bellman (HJB) dynamic programming in finding the stochastic optimal controls of interest.

Deterministic differential games in infinite horizon involving continuous and impulse controls

ABSTRACT We study a new class of two-player, zero-sum, deterministic differential games where each player uses both continuous and impulse controls in an infinite horizon with discounted payoff. We assume that the form and cost of impulses depend on nonlinear functions and the state of the system, respectively. We use Bellman's dynamic programming principle (DPP) and viscosity solutions approach to show, for this class of games, the existence and uniqueness of a solution for the associated Hamilton–Jacobi–Bellman–Isaacs (HJBI) partial differential equations (PDEs). We then, under Isaacs' condition, deduce that the lower and upper value functions coincide, and we give a computational procedure with a numerical test for the game.

Planning the Optimal Reference Flight Path of an Aircraft Using a Terrain Map

In connection with the development of aviation technology, the expansion of the functionality of modern aircraft and the complication of the tasks being solved, additional stringent requirements are imposed on the accuracy of air navigation. Along with high accuracy, navigation autonomy is also required. The autonomy of aircraft navigation implies the navigation of an aircraft without the use of active radar facilities. In this article, to calculate the optimal reference flight path of an aircraft in a horizontal plane, the Bellman dynamic programming method is proposed using a digital terrain map. A technique, algorithms have been developed, and a numerical experiment has been carried out using fragments of a digital terrain map. High accuracy and autonomy of navigation are necessary, among other things, to ensure stealth (invisibility) of aircraft by ground-based radar facilities of a potential enemy in low-altitude flight mode. The best stealth of an aircraft flight in a low-altitude flight mode is achieved not only by flying around, but also by avoiding obstacles, due to the shielding properties of the earth’s surface. Autonomy of navigation provides periodic correction of the reference flight path during long-term flight, and high accuracy prevents aircraft collisions with the earth’s surface in low-altitude flight mode. The purpose of this work is to select the optimal reference trajectory (route) of an aircraft flight in the low-altitude flight mode. This article discusses a method developed based on the principle of dynamic programming for calculating the reference flight path in the horizontal plane depending on the DEM and the implementation of the algorithm based on the proposed method with a specific numerical example. The method makes it possible to calculate the optimal flight route, which provides the greatest secrecy of the aircraft in the low-altitude flight mode using the screening properties of the earth’s surface irregularities.

Maintaining Symmetry in Optimal and Safe Control of the Ship to Avoid Collisions at Sea

The aim of this study was to make a novel symmetry analysis in relation to the importance of optimizing the ship’s trajectory and safety in situations at sea where there is a risk of collision with other ships. To achieve this, the state constraints in the optimization were formulated as ship domains generated by the neural network. In addition, the use of the Bellman dynamic programming method enabled the effective optimization of the ship’s safe control. The above assumptions were confirmed by the calculations of the optimal and safe ship traffic paths for the two valid agree with COLREGs states of visibility at sea and for different densities of the dynamic programming grid. Practical conclusions from the research were formulated, and a plan for further research on methods of ensuring safety in navigation was outlined.

Relationships between the maximum principle and dynamic programming for infinite dimensional stochastic control systems

The Pontryagin type maximum principle and Bellman's dynamic programming principle serve as two of the most important tools in solving optimal control problems. There is a huge literature on the study of relationship between them. The main purpose of this paper is to investigate the relationship between the Pontryagin type maximum principle and the dynamic programming principle for control systems governed by stochastic evolution equations in infinite dimensional space, with the control variables entering into both the drift and the diffusion terms. To do so, we first prove the dynamic programming principle for those systems without employing the martingale solutions. Then we establish the desired relationships in both cases that value function associated is smooth or nonsmooth. For the nonsmooth case, in particular, by employing the relaxed transposition solution, we discover the connection between the superdifferentials and subdifferentials of value function and the first-order and second-order adjoint equations.

Two-channel optimal discrete law of control of spacecraft with aerodynamic and inertial asymmetry during descent in Mars atmosphere

At present, spacecraft attitude stability is of great importance for both state and private space companies and agencies. In this paper, we consider a mathematical model describing perturbed motion of spacecraft as a rigid body with significant aerodynamic and inertial asymmetry relative to the center of mass in the rarefied atmosphere of Mars. The aim of this work is to obtain an approximate discrete optimized law of controlling the spacecraft attitude using Bellman's dynamic programming and averaging methods. Discrete systems of equations used in the work were solved using the Z-transform method. The reliability of the obtained control laws was confirmed by the results of numerical integration by the numerical Euler method.

Проектирование схемы системы поддержки принятия решений на основе онтологии перекомпонуемого производства

The purpose of the work is to design a scheme of the DSS of a recomposable manufacturing using an ontological approach and mathematical modeling. The decomposition of the production process in the recomposed manufacturing is carried out, the main influencing factors are identified. The conceptual scheme and hierarchy of classes of the constructed ontology of the subject area are given. In order to take into account the priority of influencing factors and identify the controlling influences affecting the speed of reconfiguration of the recomposed manufacturing, the Ishikawa cause-and-effect diagram was developed. A formalized description of the process of choosing the optimal way to reconfigure the system is proposed. A DSS scheme for the reconfigurable manufacturing has been developed, based on the consideration of temporal and spatial factors of the reconfiguration. A recomposition time structure is proposed, which allows optimizing several factors at the same time. An example of optimizing the recomposition time of a recomposable manufacturing area using the Bellman dynamic programming method and graph theory is demonstrated.

A stochastic model for the optimal allocation of hydropower flexibility in renewable energy markets

This paper considers the revenue maximization problem for a hydropower company. The company can generate excess electricity by releasing water from a reservoir and then sell it to the energy market. On the other hand, the company has an obligation to keep the reservoir level above a pre-determined level, which may require the company to purchase electricity in order to fulfill the customers’ power demand. The electricity price and reservoir level are both represented by diffusion processes. We refer to a one-factor diffusion model for electricity price, which is known to fit the data well. After applying Bellman dynamic programming principle, we derive the associated state-constrained Hamilton-Jacobi-Bellman (HJB) equation to characterize the value function. Then we prove that the value function is the viscosity solution of the state-constrained HJB equation and it is unique in this constrained optimization problem.

Optimal Control of Automatic Manipulator for Elimination of Galvanic Line Load Oscillation

This paper provides an analysis of the current state-of-the-art technologies in the field of auto-operators used in production during the electroplating process. General schemes of operations are presented, benefits and drawbacks of each scheme are discussed. The paper discusses an increase in the operating efficiency of the auto-operator in transient conditions (braking and acceleration) by reducing suspension oscillations and provides an example of a similar problem from other industries. In addition to the classification of the auto-operators, three main ways and control methods of the auto-operator of a galvanic line are presented. The main ways of eliminating oscillations during the movement of the auto-operator, as well as the rationale for the choice of adaptive (optimal) control, based on and comparing the basic control algorithms of the robot manipulator, are discussed. The comparative analysis of algorithms used to determine the optimal control has been carried out. Application field of each optimal control method described, moreover advantages and disadvantages as well as implementation methods described. Bellman dynamic programming method was chosen to eliminate oscillations of the suspension with details during the auto-operator transient conditions, the chosen method takes into account all necessary conditions to achieve the desired result.

Tether Deformation of Spinning Electrodynamic Tether System and Its Suppression with Optimal Controller

Spinning electrodynamic tether systems are considered ideal platforms for payload transportation, removal of space debris, artificial gravity, and so on, for they provide a propellantless solution to orbital maneuver and good centrifugal stability. However, all spinning electrodynamic tether systems have to transition into a spinning state from the equilibrium state, during which tethers are likely to become deformed because of Lorentz forces. This paper studies tether deformation during such a transition process. Two open-loop programs are proposed in the Lagrangian model as the reference trajectories of acceleration under different mission backgrounds. The dynamic characteristic of tether is studied in a more accurate model with distributed parameters (bead model). Considering the significant tether deformation in the case of high electrical current, an optimal controller is proposed based on Bellman dynamic programming. Numerical results indicate that the proposed control laws can ensure a safe transition of the proposed tether system into a spin and limit tether deformation to a reasonable level.

Continuity of the value function for deterministic optimal impulse control with terminal state constraint

Deterministic optimal impulse control problem with terminal state constraint is considered. Due to the appearance of the terminal state constraint, the value function might be discontinuous in general. The main contribution of this paper is the introduction of an intrinsic condition under which the value function is proved to be continuous. Then by a Bellman dynamic programming principle, the corresponding Hamilton-Jacobi-Bellman type quasi-variational inequality (QVI, for short) is derived. The value function is proved to be a viscosity solution to such a QVI. The issue of whether the value function is characterized as the unique viscosity solution to this QVI is carefully addressed and the answer is left open challengingly.

Approximate optimal method for controlling the angular motion of a spacecraft as part of an orbital tether system

An approximate optimal method for controlling the movement relative to the center of mass of a spacecraft as part of an orbital tether system is considered. The purpose of the control is to stabilize the spatial angular movement of the spacecraft relative to the direction of the tether connecting it to another spacecraft. If there is no control, the angular movement of the spacecraft relative to the direction of the tether may be unstable. This is primarily due to the disturbances that occur during the formation (deployment) of tether systems, as well as the internal small mass asymmetry of the spacecraft that inevitably occurs during its manufacture. An approximate optimal method for stabilizing the relative motion of a spacecraft on a tether is proposed, based on the Bellman dynamic programming principle and the averaging method. The averaging method is used for approximate solution of the Bellman partial differential equation, which allows in many cases to obtain the regulator’s equations in analytical form. An example of the application of this approach to stabilize the movement of a small spacecraft when deploying an orbital tether system is given.

Stochastic recursive optimal control problem with mixed delay under viscosity solution's framework

SummaryThis article is concerned with the stochastic recursive optimal control problem with mixed delay. The connection between Pontryagin's maximum principle and Bellman's dynamic programming principle is discussed. Without containing any derivatives of the value function, relations among the adjoint processes and the value function are investigated by employing the notions of super‐ and sub‐jets introduced in defining the viscosity solutions. Stochastic verification theorem is also given to verify whether a given admissible control is really optimal.

Dynamic Priority-Based Service Resource Allocation for Context-Aware Conflict Resolution in Wisdom Network with Fog Computing

With the development of wisdom network, this paper assumes that intelligent devices become more and more intelligent, which can easily collect and provide a variety of context awareness data. The research goal is to design a dynamic conflict resolution strategy for context-aware resource allocation. The limited availability of resources inevitably leads to conflicts. Considering the characteristics of wisdom network, the quality of service when solving conflicts, a mechanism is proposed to improve the quality of services and to solve the resources allocation conflicts. This paper constructs the optimal model of context-aware based on a differential game and optimizes the resource allocation of context-aware based on the priority of scenarios. Fog computing is used to provide enough computing resources for the control of resource allocation of context-aware. The Bellman dynamic programming is introduced to solve the feedback Nash equilibrium solution of the proposed differential game model, to obtain the optimal allocation of service resources and solve the effectiveness of resource allocation.

Bellman Filtering for State-Space Models

This article presents a filter for state-space models based on Bellman's dynamic programming principle applied to the mode estimator. The proposed Bellman filter generalises the Kalman filter including its extended and iterated versions, while remaining equally inexpensive computationally. The Bellman filter is also (unlike the Kalman filter) robust under heavy-tailed observation noise and applicable to a wider range of (nonlinear and non-Gaussian) models, involving e.g. count, intensity, duration, volatility and dependence. The Bellman-filtered states are shown to be convergent, in quadratic mean, towards a small region around the true state. (Hyper)parameters are estimated by numerically maximising a filter-implied log-likelihood decomposition, which is an alternative to the classic prediction-error decomposition for linear Gaussian models. Simulation studies reveal that the Bellman filter performs on par with (or even outperforms) state-of-the-art simulation-based techniques, e.g. particle filters and importance samplers, while requiring a fraction (e.g. 1%) of the computational cost, being straightforward to implement and offering full scalability to higher dimensional state spaces.

Про проблему стабілізації керованих стохастичних дифереціально-функціональних систем із скінченним запізненням

Works [1–4] analyze a problem of stabilization of systems, which are defined by a stochastic differential-functional equations with impulsive Markov disturbances with a constant or finite delay, in presence of a transitional process and a delay at the same time. This work considers a more generalized problem of stabilization of the control stochastic differential-functional systems with finite delay and mutually independent Wiener processes. The delay is constructed on the space of the Skorokhod of right continuous functions with left limits [1]. This systems must be asymptotically stable by the probability and provide preassigned optimivity of a transient process. The control be selected is built on the principle of inverse communication, obtained as a Markov process [3, 4]. The problem of optimal stabilization is considered in the context of the given quality criteria, builds on Bellman's dynamic programming principles. The first part of the work analyzes the properties of Markov processes. The corresponding lemma is formulated as a result. In the second part obtained the infinitesimal operator of the corresponding Markov process, is formulated and the basic theorem of stabilization is proved. The proof algorithm is based on using the Ito-formula. Examples of use are given. In the third part an optimal stabilization algorithm has been demonstrated to investigate a linear systems. For the case of linear systems, the stabilization theorem is formulated. The results of the scientific research were obtained for use in technical systems. The results obtained and the arguments given are valid in the determined case as well. This work is part one of the first scientific research, the second part will contain more examples and use the method of successive approximations.

Resource Allocation in Unmanned Aerial Vehicle (UAV)-Assisted Wireless-Powered Internet of Things.

Most of the wireless nodes in the Internet of Things (IoT) environment face the limited energy problem and the way to provide a sustainable energy for these nodes has become an urgent challenge. In this paper, we present an unmanned aerial vehicle (UAV) to power the wireless nodes in the IoT and an investigation on the optimal resource allocation approach based on dynamic game theory. This IoT system consists of one UAV as the power source and information receiver. The wireless nodes can be powered and collected by the UAV. In order to distinguish the wireless nodes, the wireless nodes are divided into two categories based on the energy consumption. The UAV tries to power these two categories of nodes using a different power level based on the proposed approach, where the wireless nodes control the resources for information transmission. Based on Bellman dynamic programming, the optimal allocated resources for power transfer and information transmission are obtained for both the UAV and wireless nodes, respectively. In order to show the effectiveness of the proposed model and approach, we present numerical simulations.