Reflections on optimality and dynamic programming

Abstract

Discrete models and continuous control systems are considered in regard to optimality of their trajectories. Some aspects of the principle of optimality [1, p. 83] are analyzed, and it is shown to imply total optimality, that is, the optimality of every part of an optimal trajectory. Certain autonomous systems with free admissible variations possess this property. Nonautonomous optimal systems are not, in general, totally optimal, in which case the principle of optimality is not valid. A modification is proposed for the derivation of the main functional equation to demonstrate that dynamic programming and its functional equations are valid also in the case of nonoptimal remaining trajectories under a certain contiguity condition that is defined and analyzed in the paper. Control systems with incomplete information or structural limitations on controls do not, in general, satisfy the contiguity condition. Control problems for such systems may have optimal solutions which, however, cannot be obtained by dynamic programming. This fact is shown in an example of a widely used engineering system for which an optimal trajectory has all its remaining parts nonoptimal and noncontiguous to the optimal trajectory. The paper presents theoretical justification of dynamic programming for contiguous systems that do not conform to the principle of optimality. Examples are presented to illustrate the results which open new avenues in modeling and optimization of general (not totally optimal) control systems.