Bellman's Principle Research Articles

Optimal protocol for a collective flashing ratchet

We study a system of independent Brownian particles in a flashing ratchet potential that can be turned on or off depending on the position of the particles, with the aim of maximising the speed of the center of mass in the long run. First, an explanation of how to find the optimal protocol using Bellman's principle of optimality for any number of particles is given. Then the problem is numerically solved for a system of 2 particles. Simulations show that the optimal protocol performs better than the maximisation of instantaneous center-of-mass speed, a protocol known to give better results for 2 particles than any open-loop protocol.

Minimum Time Kinematic Motions of a Cartesian Mobile Manipulator for a Fruit Harvesting Robot

This paper describes an analytical procedure to calculate the time-optimal trajectory for a mobile Cartesian manipulator to traverse between any two fruits it picks up it. The goal is to minimize the time required from the retrieval of one fruit to that of the next while adhering to velocity, acceleration, location, and endpoint constraints. This is accomplished using a six stage procedure, based on Bellman's Principle of Optimality and nonsmooth optimization that is completely analytical and requires no numerical computations. The procedure sequentially calculates all relevant parameters, from which side of the mobile platform to place the fruit on to the velocity profile and drop-off point, that yield a minimum time trajectory. In addition, it provides a time window under which the mobile manipulator can traverse from any fruit to any other, which can be used for a globally optimal retrieving sequence algorithm.

An optimal execution problem with market impact

We study an optimal execution problem in a continuous-time market model that considers market impact. We formulate the problem as a stochastic control problem and investigate properties of the corresponding value function. We find that right-continuity at the time origin is associated with the strength of market impact for large sales, otherwise the value function is continuous. Moreover, we show the semi-group property (Bellman principle) and characterise the value function as a viscosity solution of the corresponding Hamilton-Jacobi-Bellman equation. We introduce some examples where the forms of the optimal strategies change completely, depending on the amount of the trader's security holdings and where optimal strategies in the Black-Scholes type market with nonlinear market impact are not block liquidation but gradual liquidation, even when the trader is risk-neutral.

Formulation of an Optimal Execution Problem with Market Impact: Derivation from Discrete-Time Models to Continuous-Time Models

We study an optimal execution problem in a continuous-time market model that considers market impact. We formulate the problem as a stochastic control problem and investigate properties of the corresponding value function. We find that right-continuity at the time origin is associated with the strength of market impact for large sales, otherwise the value function is continuous. Moreover, we show the semi-group property (Bellman principle) and characterise the value function as a viscosity solution of the corresponding Hamilton-Jacobi-Bellman equation. We introduce some examples where the forms of the optimal strategies change completely, depending on the amount of the trader's security holdings and where optimal strategies in the Black-Scholes type market with nonlinear market impact are not block liquidation but gradual liquidation, even when the trader is risk-neutral.

Verification by Stochastic Perron's Method in stochastic exit time control problems

We apply the Stochastic Perron Method, created by Bayraktar and Sîrbu, to a stochastic exit time control problem. Our main assumption is the validity of the Strong Comparison Result for the related Hamilton–Jacobi–Bellman (HJB) equation. Without relying on Bellman's optimality principle we prove that inside the domain the value function is continuous and coincides with a viscosity solution of the Dirichlet boundary value problem for the HJB equation.

Optimal control of uncertain quantized linear discrete‐time systems

SummaryIn this paper, the adaptive optimal regulator design for unknown quantized linear discrete‐time control systems over fixed finite time is introduced. First, to mitigate the quantization error from input and state quantization, dynamic quantizer with time‐varying step‐size is utilized wherein it is shown that the quantization error will decrease overtime thus overcoming the drawback of the traditional uniform quantizer. Next, to relax the knowledge of system dynamics and achieve optimality, the adaptive dynamic programming methodology is adopted under Bellman's principle by using quantized state and input vector. Because of the time‐dependency nature of finite horizon, an adaptive online estimator, which learns a newly defined time‐varying action‐dependent value function, is updated at each time step so that policy and/or value iterations are not needed. Further, an additional error term corresponding to the terminal constraint is defined and minimized along the system trajectory. The proposed design scheme yields a forward‐in‐time and online scheme, which enjoys great practical merits. Lyapunov analysis is used to show the boundedness of the closed‐loop system; whereas when the time horizon is stretched to infinity as in the case of infinite horizon, asymptotic stability of the closed‐loop system is demonstrated. Simulation results on a benchmarking batch reactor system are included to verify the theoretical claims. The net result is the design of the optimal adaptive controller for uncertain quantized linear discrete‐time systems in a forward‐in‐time manner. Copyright © 2014 John Wiley & Sons, Ltd.

Stochastic Maximum Principle for Mean-Field Type Optimal Control Under Partial Information

This technical note is concerned with a partially observed optimal control problem, whose novel feature is that the cost functional is of mean-field type. Hence determining the optimal control is time inconsistent in the sense that Bellman's dynamic programming principle does not hold. A maximum principle is established using Girsanov's theorem and convex variation. Some nonlinear filtering results for backward stochastic differential equations (BSDEs) are developed by expressing the solutions of the BSDEs as some Itô's processes. An illustrative example is demonstrated in terms of the maximum principle and the filtering.

Research of inverse mathematical model to high-speed trains

Operation safety and stability of the train mainly depend on the interaction between the wheel and rail. Knowledge of wheel/rail contact force is important for vehicle control systems that aim to enhance vehicle stability and passenger safety. Since wheel/rail contact forces of high-speed train are very difficult to measure directly, a new estimation process for wheel/rail contact forces was introduced in this work. Based on the state space equation, dynamic programming methods and the Bellman principle of optimality, the main theoretical derivation of the inversion mathematical model was given. The new method overcomes the weakness of large fluctuations which exist in current inverse techniques. High-speed vehicle was chosen as the research object, accelerations of axle box as input conditions, 10 degrees of freedom vertical vibration model and 17 degrees of freedom lateral vibration model were established, respectively. Under 250 km/h, the vertical and lateral wheel/rail forces were identified. From the time domain and frequency domain, the comparison of the results between inverse and SIMPACK models were given. The results show that the inverse mathematical model has high precision for inversing the wheel/rail contact forces of an operation high-speed vehicle.

General Linear Quadratic Optimal Stochastic Control Problem Driven by a Brownian Motion and a Poisson Random Martingale Measure with Random Coefficients

In this article, we consider a linear-quadratic optimal control problem (LQ problem) for a controlled linear stochastic differential equation driven by a multidimensional Browinan motion and a Poisson random martingale measure in the general case, where the coefficients are allowed to be predictable processes or random matrices. By the duality technique, the dual characterization of the optimal control is derived by the optimality system (so-called stochastic Hamilton system), which turns out to be a linear fully coupled forward-backward stochastic differential equation with jumps. Using a decoupling technique, the connection between the stochastic Hamilton system and the associated Riccati equation is established. As a result, the state feedback representation is obtained for the optimal control. As the coefficients for the LQ problem are random, here, the associated Riccati equation is a highly nonlinear backward stochastic differential equation (BSDE) with jumps, where the generator depends on the unknown variables K, L, and H in a quadratic way (see (5.9) herein). For the case where the generator is bounded and is linearly dependent on the unknown martingale terms L and H, the existence and uniqueness of the solution for the associated Riccati equation are established by Bellman's principle of quasi-linearization.

Dynamic Phasor Estimates Under the Bellman's Principle of Optimality: The Taylor-LQG-Fourier Filters

Recently Taylor <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">K</sup> -Kalman-Fourier filters were proposed for estimating dynamic phasors to provide instantaneous estimates and drastically reduce the total vector error by a factor of 10. However, they exhibit resonant frequencies at the edges of the pass band, and high-interharmonic gains. In this paper, the optimal linear quadratic (LQ) control is applied to design feedback filters referred to as Taylor <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">K</sup> -LQG-Fourier filters. This method reduces the interharmonic gains and the resonant frequency at the passband edges of the Taylor <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">K</sup> -Kalman-Fourier filter. The estimates from oscillating signals obtained through this optimal technique are quasi-instantaneous, and provide estimates of the instantaneous frequency, and its rate of change, preserving its synchrony with the signal for control applications. The effectiveness of the proposed algorithm is verified through simulations and experimental results.

The equivalence of viscosity and distributional subsolutions for convex subequations — a strong Bellman principle

There are two useful ways to extend nonlinear partial differential inequalities of second order beyond the class of C 2 functions: one uses viscosity theory and the other uses the theory of distributions. This paper considers the convex situation where both extensions can be applied. The main result is that under a natural “second-order completeness” hypothesis, the two sets of extensons are isomorphic, in a sense that is made precise.

Finding Reliable Shortest Path in Stochastic Time-dependent Network

This paper addresses the problem of finding reliable a priori shortest path to maximize the probability of arriving on time in a stochastic and time-dependent network. Optimal solutions to the problem can be obtained from finding non-dominated paths, which are defined based on first-order stochastic dominance. We formulate the problem of finding non-dominated paths as a general dynamic programming problem because Bellman's principle of optimality can be applied to construct non-dominated paths. A label-correcting algorithm is designed to find optimal paths based on the new proporty for which Bellman's Principle holds. Numerical results are provided using a small network.

A COMPROMISE PROGRAMMING APPROACH TO MULTIOBJECTIVE MARKOV DECISION PROCESSES

A Markov decision process (MDP) is a general model for solving planning problems under uncertainty. It has been extended to multiobjective MDP to address multicriteria or multiagent problems in which the value of a decision must be evaluated according to several viewpoints, sometimes conflicting. Although most of the studies concentrate on the determination of the set of Pareto-optimal policies, we focus here on a more specialized problem that concerns the direct determination of policies achieving well-balanced tradeoffs. To this end, we introduce a reference point method based on the optimization of a weighted ordered weighted average (WOWA) of individual disachievements. We show that the resulting notion of optimal policy does not satisfy the Bellman principle and depends on the initial state. To overcome these difficulties, we propose a solution method based on a linear programming (LP) reformulation of the problem. Finally, we illustrate the feasibility of the proposed method on two types of planning problems under uncertainty arising in navigation of an autonomous agent and in inventory management.

OPTIMISTIC VALUE MODEL OF UNCERTAIN OPTIMAL CONTROL

Optimal control is an important field of study both in theory and in applications. Based on uncertainty theory, an expected value model of uncertain optimal control problem was studied by Zhu. In this paper, an optimistic value model for uncertain optimal control problem is investigated. Applying Bellman's principle of optimality, the principle of optimality for the model is presented. And then the equation of optimality is obtained for the optimistic value model of uncertain optimal control. Finally, a portfolio selection problem is solved by this equation of optimality.

The Optimization of Microgrids Operation through a Heuristic Energy Management Algorithm

The concept of microgrid was first introduced in 2001 as a solution for reliable integration of distributed generation and for harnessing their multiple advantages. Specific control and energy management systems must be designed for the microgrid operation in order to ensure reliable, secure and economical operation; either in grid-connected or stand-alone operating mode. The problem of energy management in microgrids consists of finding the optimal or near optimal unit commitment and dispatch of the available sources and energy storage systems so that certain selected criteria are achieved. In most cases, energy management problem do not satisfy the Bellman's principle of optimality because of the energy storage systems. Consequently, in this paper, an original fast heuristic algorithm for the energy management on stand-alone microgrids, which avoids wastage of the existing renewable potential at each time interval, is presented. A typical test microgrid has been analysed in order to demonstrate the accuracy and the promptness of the proposed algorithm. The obtained cost of energy is low (the quality of the solution is high), the primary adjustment reserve is correspondingly assured by the energy storage system and the execution runtime is very short (a fast algorithm). Furthermore, the proposed algorithm can be used for real-time energy management systems.

Testing for multiple change points

In this paper we concentrate on testing for multiple changes in the mean of a series of independent random variables. Suggested method applies a maximum type test statistic. Our primary focus is on an effective calculation of critical values for very large sample sizes comprising (tens of) thousands of observations and a moderate to large number of segments. To that end, Monte Carlo simulations and a modified Bellman's principle of optimality are used. It is shown that, indisputably, computer memory becomes a critical bottleneck in solving a problem of such a size. Thus, minimization of the memory requirements and appropriate order of calculations appear to be the keys to success. In addition, the formula that can be used to get approximate asymptotic critical values using the theory of exceedance probability of Gaussian fields over a high level is presented.

Adaptive Estimation of Time-Varying Sparse Signals

We consider the problem of adaptively designing compressive measurement matrices for estimating time-varying sparse signals. We formulate this problem as a partially observable Markov decision process. This formulation allows us to use Bellman's principle of optimality in the implementation of multi-step lookahead designs of compressive measurements. We compare the performance of adaptive versus traditional non-adaptive designs and study the value of multi-step (non-myopic) versus one-step (myopic) lookahead adaptive schemes by introducing two variations of the compressive measurement design problem. In the first variation, we consider the problem of sequentially selecting measurement matrices with fixed dimensions from a prespecified library of measurement matrices. In the second variation, the number of compressive measurements, i.e., the number of rows of the measurement matrix, is adaptively chosen. Once the number of measurements is determined, the matrix entries are chosen according to a prespecified adaptive scheme. Each of these two problems is judged by a separate performance criterion. The gauge of efficiency in the first problem is the conditional mutual information between the sparse signal support and measurements. The second problem applies a linear combination of the number of measurements and conditional mutual information as the performance measure. Through several simulations, we study the effectiveness of different designs in various settings. The primary focus in these simulations is the application of a method known as rollout. However, the computational load required for using the rollout method has also inspired us to adapt two data association heuristics to the compressive sensing paradigm. These heuristics show promising decreases in the amount of computation for propagating distributions and searching for optimal solutions.

Existence, uniqueness and removable singularities for nonlinear partial differential equations in geometry

This paper surveys some recent results on existence, uniqueness and removable singularities for fully nonlinear differential equations on manifolds. The discussion also treats restriction theorems and the strong Bellman principle.

Interactive procedure for a multiobjective stochastic discrete dynamic problem

Multiple objectives and dynamics characterize many sequential decision problems. In the paper we consider returns in partially ordered criteria space as a way of generalization of single criterion dynamic programming models to multiobjective case. In our problem evaluations of alternatives with respect to criteria are represented by distribution functions. Thus, the overall comparison of two alternatives is equivalent to the comparison of two vectors of probability distributions. We assume that the decision maker tries to find a solution preferred to all other solutions (the most preferred solution). In the paper a new interactive procedure for stochastic, dynamic multiple criteria decision making problem is proposed. The procedure consists of two steps. First, the Bellman principle is used to identify the set of efficient solutions. Next interactive approach is employed to find the most preferred solution. A numerical example and a real-world application are presented to illustrate the applicability of the proposed technique.

The possibility of mineral deposit exploitation control with Bellman's optimality principle use – a Polkowice–Sieroszowice copper mine case study

Decision-making problems in mineral deposit management can be solved with mathematical techniques. Accomplishment of geological mining projects is usually long lasting and expensive as well as it proceeds in uncertainty conditions and is burdened with a considerable risk. The article presents possible applications of Bellman's optimality principle in exploitation control in the Polkowice–Sieroszowice copper mine. Exploitation blocks of a certain area in the mine differ mutually in ore quality and geological mining conditions. The blocks can be exploited in a certain sequence, which at optimal financial means allocation guarantees maximization of the incomes. In the first step of solving the decision-making problem the price of ore has been calculated. In the second the optimality equations have been constructed. These equations describe possible to achieve incomes related to allocated and consumed financial means. Mathematically solving the equations has allowed selecting the sequence of optimal decisions constituting the optimal strategy.