Optimal Impulsive Control Problems Research Articles

Optimal impulse control problems with time delays: An illustrative example

For impulse control systems described by a measure driven differential equation, depending linearly on the measure, it is customary to interpret the state trajectory corresponding to an impulse control, specified by a measure, as the limit of state trajectories associated with some sequence of conventional controls approximating the measure. It is known that, when the measure is vector valued, it is possible that different choices of approximating sequences for the measure give rise to different limiting state trajectories. If the measure is scalar valued, however, there is a unique limiting trajectory. Now consider impulse control systems, in which the right side of the measure driven differential equation depends on both the current and delayed states. In recent work by the authors it has been shown that, for such impulse control systems with time delay, the state trajectory corresponding to a given measure may be non-unique, even when the measure is scalar valued. It was also shown that each limiting state trajectory can be identified with the unique state trajectory associated with some measure together with a family of ‘attached controls’. (The attached controls capture the nature of the measure approximation.) The authors also derived a maximum principle governing minimizers for a general class of impulse optimal control problems with time delay, in which the domain of the optimization problem comprises measures coupled with a family of ‘attached controls’. The purpose of this paper is both to illustrate, by means of an example, this newly discovered non-uniqueness phenomenon and to provide the first application of the new maximum principle, to investigate minimizers for scalar input impulse optimal control problems with time delay, in circumstances when limiting state trajectories associated with a given measure control are not unique. The example is an optimal control problem, for which the underlying control system is a forced harmonic oscillator, with scalar impulse control, in which the control gain is a nonlinear function of the current and delayed states.

Impulsive optimal control problems with time delays in the drift term

Impulsive optimal control problems with time delays in the drift term

Approximate Solution of Optimal Pulse Control Problem Associated with the Heat Conduction Process

Abstract We consider the approximate solution of the control problem with minimum energy for an object described by the heat equation, with the process described by the linear equation of parabolic type and the system controlled by impulsive external influences. Our optimal control problem deals with finding a control parameter belonging to the class of admissible controls that provides the desired temperature distribution in a finite time with minimal energy consumption (energy consumption is described by the quadratic functional). Previous works dedicated to optimal impulse control problems have mostly used the Pontryagin’s maximum principle. However, from a practical point of view, this approach does not lead to satisfactory results. This is due to the fact that the corresponding boundary value problems in this case have no solution in a traditional class of absolutely continuous trajectories. In this work, we propose a method based on the moment relations. We seek for the approximate solution of the corresponding boundary value problem in the form of finite Fourier sum and state our optimal control problem in a finite-dimensional phase space. As a result, we obtain an optimal impulse control problem in a finite-dimensional function space. Taking into account the given condition for a finite time, we reduce the obtained problem to the L-problem of moments. Thus, the problem of finding a control parameter is reduced to the solution of the system of Fredholm integral equations of the first kind, with the norm of the sought solution not exceeding a given number. By Levi’s theorem, every element of Hilbert space can be represented by the sum of the elements of two orthogonal subspaces. This assertion makes it possible to find control parameters in analytical form. We also establish the convergence of the chosen approximation.

Optimal stopping and impulse control in the presence of an anticipated regime switch

We consider a class of stochastic optimal stopping and impulse control problems where the agent solving the problem anticipates that a regime switch will happen at a random time in the future. We assume that there are only two regimes, the regime switching time is exponentially distributed, the underlying stochastic process is a linear, regular, time-homogeneous diffusion in both regimes and the payoff may be regime-dependent. This is in contrast with most existing literature on the topic, where regime switching is modulated by a continuous-time Markov chain and the underlying process and payoff belong to the same parametric family in all regimes. We state a set of easily verifiable sufficient conditions under which the solutions to these problems are given by one-sided threshold strategies. We prove uniqueness of the thresholds and characterize them as solutions to certain algebraic equations. We also study how anticipation affects optimal policies i.e. we present various comparison results for problems with and without regime switching. It may happen that the anticipative value functions and optimal policies coincide with the usual ones even if the regime switching structure is non-trivial. We illustrate our results with practical examples.

An Efficient Impulsive Adaptive Dynamic Programming Algorithm for Stochastic Systems.

In this study, a novel general impulsive transition matrix is defined, which can reveal the transition dynamics and probability distribution evolution patterns for all system states between two impulsive "events," instead of two regular time indexes. Based on this general matrix, the policy iteration-based impulsive adaptive dynamic programming (IADP) algorithm along with its variant, which is a more efficient IADP (EIADP) algorithm, are developed in order to solve the optimal impulsive control problems of discrete stochastic systems. Through analyzing the monotonicity, stability, and convergency properties of the obtained iterative value functions and control laws, it is proved that the IADP and EIADP algorithms both converge to the optimal impulsive performance index function. By dividing the whole impulsive policy into smaller pieces, the proposed EIADP algorithm updates the iterative policies in a "piece-by-piece" manner according to the actual hardware constraints. This feature of the EIADP method enables these ADP-based algorithms to be fully optimized to run on all "sizes" of computing devices including the ones with low memory spaces. A simulation experiment is conducted to validate the effectiveness of the present methods.

Robust Optimal Impulsive Blood Glucose Control Exploiting a Direct Searching Algorithm

In spite of numerous developments in blood glucose control based on the artificial pancreas, multiple daily injections (MDIs) are still the most applicable method in insulin-dependent diabetes mellitus (IDDM) subjects. These patients measure their blood glucose level (BGL) regularly and inject insulin in multiple dosages. In the present study, an efficient, robust optimal impulsive control approach is introduced to attain a safe range of blood glucose considering everyday life factors, including exercise, food intake, and circadian rhythms. This article assumes the dosages of the MDIs as control design parameters and addresses an appropriate approach for optimal dosage determination. The BGL regulation problem by the MDI program is characterized as a robust optimal impulsive control problem with uncertainties, and it is required to minimize the BGL error as the objective function. To achieve this, the polytopic form of the nonlinear glucose-insulin system is derived, and the feasible region for MDI is determined using a direct searching algorithm. Then, the optimal MDI is found by a sequential quadratic programming (SQP) approach. Regarding several uncertainties and the impulsive nature of insulin injection dosages, the proposed approach efficiently solves the BGL regulation problem. The proposed approach is evaluated by a simulated model of the blood-glucose insulin system in two regular and overfeeding scenarios with (mild, moderate) and without physical activity. The results show that the proposed method reduces the amount of injected insulin from 14% to 20% compared to a state-of-the-art approach and, simultaneously, reduces the severe hyperglycemia risk from 15% to 25% for overeating in three injection scenarios.

Spiking Adaptive Dynamic Programming Based on Poisson Process for Discrete-Time Nonlinear Systems.

In this article, a new iterative spiking adaptive dynamic programming (SADP) method based on the Poisson process is developed to solve optimal impulsive control problems. For a fixed time interval, combining the Poisson process and the maximum likelihood estimation (MLE), the three-tuple of state, spiking interval, and probability of Poisson distribution can be computed, and then, the iterative value functions and iterative control laws can be obtained. A property analysis method is developed to show that the value functions converge to optimal performance index function as the iterative index increases from zero to infinity. Finally, two simulation examples are given to verify the effectiveness of the developed algorithm.

Robust Space Trajectory Design Using Belief Optimal Control

This paper presents a novel approach to the robust solution of optimal impulsive control problems under aleatory and epistemic uncertainty. The novel approach uses belief Markov decision processes to reformulate the control problem in terms of uncertainty distributions, called beliefs, rather than the realizations of the system states. This formulation leads to the definition of a belief optimal control problem where the cost function and constraints are functions of the uncertainty distributions. The control formulation encompasses orbit determination arcs as well. The belief optimization is solved with a shooting-like transcription and a nonlinear programming solver to optimize the resulting discretized problem. Both aleatory and epistemic uncertainties are propagated with a nonintrusive polynomial expansion to capture the nonlinearities of the dynamics. Finally, this new approach is applied to the robust optimization of a flyby trajectory of the Europa Clipper mission in a scenario characterized by knowledge, execution, and observation uncertainty.

Hybrid optimal impulse control

This paper is concerned with an optimal impulse control problem under a hybrid diffusion (or, regime switching) model, where the state of the system consists of a number of diffusions coupled by a continuous-time finite-state Markov chain. The objective is to minimize the expected discounted cost from exerting the impulse control in the infinite horizon. Based on the dynamic programming principle (DPP), the value function of the hybrid optimal impulse control problem is shown to be the unique viscosity solution to the associated Hamilton–Jacobi–Bellman (HJB) equation, which is in the form of a coupled system of variational inequalities. Moreover, a verification theorem as the sufficient condition for optimality of a solution is also established. The optimal impulse control, indicates when and how it is optimal to intervene, is described by the obstacle part of the HJB equation. Finally, the general theoretical results are applied to an optimal cash management problem. The value function in closed-form and an explicit optimal policy are obtained, which exhibit clearly the effect of regime switching on the agent’s decision.

Mathematical modeling and analysis of biological control strategy of aphid population

<abstract><p>To study the biological control strategy of aphids, in this paper we propose host-parasitoid-predator models for the interactions among aphids, parasitic wasps and aphidophagous Coccinellids incorporating impulsive releases of Coccinellids, and then study the long-term control and limited time optimal control of aphids by adjusting release amount and release timing of Coccinellids. For the long-term control, the existence and stability of the aphid-eradication periodic solution are investigated and threshold conditions about the release amount and release period to ensure the ultimate extinction of the aphid population are obtained. For the limited-time control, three different optimal impulsive control problems are studied. A time rescaling technique and an optimization algorithm based on gradient are applied, and the optimal release amounts and timings of natural enemies are gained. Our simulations indicate that in the limited-time control, the optimal selection of release timing should be given higher priority compared with the release amount.</p></abstract>

Optimal impulse control of West Nile virus

<abstract><p>We construct a West Nile virus epidemic model that includes the interaction between the bird hosts and mosquito vectors, mosquito life stages (eggs, larvae, adults), and the dynamics of both larvicide and adulticide. We derive the basic reproduction number for the epidemic as the spectral radius of the next generation matrix. We formulate two impulsive optimal control problems which seek to balance the cost of insecticide applications (both the timing and application level) with the benefit of (1) vector control: reducing the number of mosquitoes or (2) disease control: reducing the disease burden. We reformulate these impulsive optimal control problems as nonlinear optimization problems and derive associated necessary conditions for the optimal controls. Numerical simulations are used to address three questions: How does the control and its impact on the system vary with the objective type? Is it beneficial to optimize the treatment timing? How does the control and its impact on the population vary with the type of pesticide used?</p></abstract>

Aggregated occupation measures and linear programming approach to constrained impulse control problems

For a constrained optimal impulse control problem of an abstract dynamical system, we introduce the occupation measures along with the aggregated occupation measures and present two associated linear programs. We prove that the two linear programs are equivalent under appropriate conditions, and each linear program gives rise to an optimal strategy in the original impulse control problem. In particular, we show the absence of the relaxation gap. By means of an example, we also present a detailed comparison of the occupation measures and linear programs introduced here with the related notions in the literature.

Оптимизация импульсных управляемых систем с промежуточными фазовыми ограничениями

In this paper, we consider an optimal impulsive control problem with intermediate state constraints. The peculiarity of the problem consists in a non-standard way of specifying of intermediate constraints. So the constraints must be satisfied for at least one selection of a set-valued solution to the impulsive control system. We prove a theorem for the existence of an optimal control and propose the reduction procedure that transforms the initial optimal control problem with intermediate constraints into a hybrid problem with control parameters. This hybrid problem gives an equivalent description of the optimal impulsive control problem. We discuss a numerical algorithm based on a direct collocation method and give a schema to the corresponding numerical calculations for a test example.

Continuity of the value function for deterministic optimal impulse control with terminal state constraint

Deterministic optimal impulse control problem with terminal state constraint is considered. Due to the appearance of the terminal state constraint, the value function might be discontinuous in general. The main contribution of this paper is the introduction of an intrinsic condition under which the value function is proved to be continuous. Then by a Bellman dynamic programming principle, the corresponding Hamilton-Jacobi-Bellman type quasi-variational inequality (QVI, for short) is derived. The value function is proved to be a viscosity solution to such a QVI. The issue of whether the value function is characterized as the unique viscosity solution to this QVI is carefully addressed and the answer is left open challengingly.

Numerical Algorithms For State-Linear Optimal Impulsive Control Problems Based On Feedback Necessary Optimality Conditions

We propose and compare three numeric algorithms for optimal control of state-linear impulsive systems. The algorithms rely on the standard transformation of impulsive control problems through the discontinuous time rescaling, and the so-called “feedback”, direct and dual, maximum principles. The feedback maximum principles are variational necessary optimality conditions operating with feedback controls, which are designed through the usual constructions of the Pontryagin’s Maximum Principle (PMP); though these optimality conditions are formulated completely in the formalism of PMP, they essentially strengthen it. All the algorithms are non-local in the sense that they are aimed at improving non-optimal extrema of PMP (local minima), and, therefore, show the potential of global optimization.

Linear programming approach to optimal impulse control problems with functional constraints

This paper considers an optimal impulse control problem of dynamical systems generated by a flow. The performance criteria are total costs over the infinite time horizon. Apart from the main performance to be minimized, there are multiple constraints on performance functionals of a similar type. Under a natural set of compactness-continuity conditions on the system primitives, we establish a linear programming approach, and prove the existence of a stationary optimal control strategy out of a more general class of randomized strategies. This is done by making use of the tools from Markov decision processes.

Optimal Impulsive Control Using Adaptive Dynamic Programming and its Application in Spacecraft Rendezvous.

Optimal control of nonlinear impulsive systems with free impulse instants and the number of impulses is investigated in this study. A scheme based on adaptive dynamic programming is developed, which leads to a feedback (approximate) solution to the defined optimal impulsive control problem. This is done by proposing a learning algorithm for tuning parameters of a function approximator, which, once tuned offline, provides feedback solution on-the-fly. The scheme is shown to handle single and multiple impulsive actuators with a small online computational burden. Afterward, the controller is applied to a challenging problem, namely, the orbital maneuver of spacecraft with the fixed final time using impulsive actuators. The objective is triggering the actuators in a fuel-optimal manner such that the spacecraft transfers to the desired orbit at a prescribed time. It was shown that the proposed scheme leads to simultaneous and feedback path planning and control for the maneuver. The potentials of the scheme are analyzed in different scenarios, including enforcing a shorter final time, selecting different initial states, and incorporating actuator faults.

Optimal Impulsive Control Problems Motivated by Mechanical Systems With Vibrations and Blockable DOFs

We consider singular optimal control problems stated for differential-algebraic measure-driven equations. The latter take the form of complementarity systems, which admit jumps of state trajectories produced by open-loop and state-conditioned impulses, acting independently. As an inspiration for the addressed class of models, we turn to mechanical systems driven by time-dependent holonomic (shock impacts, control vibrations) and/or nonholonomic (impactive blocking/releasing system's degrees of freedom) constraints. For our problems, we adapt an approach based on singular space-time transformation. This approach involves approximation of discontinuous solutions of the impulsive complementarity system by ordinary, physically meaningful control processes. Since the complexity of the addressed problems is too high for the analytical investigation, we focus on their numerical investigation, and propose a general computational method.

On the equivalence of the integral and differential Bellman equations in impulse control problems

When solving optimal impulse control problems, one can use the dynamic programming approach in two different ways: at each time moment, one can make the decision whether to apply a particular type of impulse, leading to the instantaneous change of the state, or apply no impulses at all; or, otherwise, one can plan an impulse after a certain interval, so that the length of that interval is to be optimised along with the type of that impulse. The first method leads to the differential Bellman equation, while the second method leads to the integral Bellman equation. The target of the current article is to prove the equivalence of those Bellman equations in many specific models. Those include abstract dynamical systems, controlled ordinary differential equations, piece-wise deterministic Markov processes and continuous-time Markov decision processes.

Normality and Nondegeneracy of the Maximum Principle in Optimal Impulsive Control Under State Constraints

We investigate nondegenerate and normal forms of the maximum principle for general, free end-time, impulsive optimal control problems with state and endpoint constraints. We introduce constraint qualifications sufficient to avoid degeneracy or abnormality phenomena, which do not require any convexity and impose the existence of an inward pointing velocity just on the subset of times, in which the extended optimal trajectory has an outward pointing velocity (w.r.t. the state constraint). These conditions extend to impulsive problems some conditions, recently proposed by F. Fontes and H. Frankowska, for conventional optimization problems. The nontriviality of this extension is illustrated through some examples.