Continuous Value Function Research Articles

Reinforcement learning control with function approximation via multivariate simplex splines

SummaryIn the field of optimal control for continuous nonlinear systems, function approximation methods are often employed to overcome the curse of dimensionality. Compared to other global function approximators like neural networks, multivariate splines can be easily evaluated and adapted on a local basis with linearity in the parameters. In this work, a multivariate spline based reinforcement learning (RL) strategy is proposed for solving the continuous‐time nonlinear control problem. Based on the classic value iteration method, multivariate splines are integrated into RL algorithms to approximate continuous value functions and policy functions from discrete action and value samples. Hence, the determined splines with updated coefficients can be utilized in continuous control of nonlinear systems. In the simulation experiment, the performance of the spline‐based RL control is evaluated in controlling an under‐actuated inverted pendulum. The proposed method is compared with the value iteration based discrete control strategy and the neural network based continuous control strategy. The simulation results indicate that the proposed method based on multivariate splines has better control performance with less state oscillations, energy consumption and convergence time in comparison with discrete value iteration and neural network based RL, and the adoption of simplex splines improves the function approximation efficiency with less computation time than neural network optimization.

A Note on Lagrange Interpolation of |x| on the Chebyshev and Chebyshev–Lobatto Nodal Systems: The Even Cases

Throughout this study, we continue the analysis of a recently found out Gibbs–Wilbraham phenomenon, being related to the behavior of the Lagrange interpolation polynomials of the continuous absolute value function. Our study establishes the error of the Lagrange polynomial interpolants of the function |x| on [−1,1], using Chebyshev and Chebyshev–Lobatto nodal systems with an even number of points. Moreover, with respect to the odd cases, relevant changes in the shape and the extrema of the error are given.

Stochastic optimal control in infinite dimensions with state constraints

We consider the state-constrained stochastic optimal control problem in infinite-dimensional separable Hilbert spaces, where the state process is driven by the Q-Wiener process and the (possibly unbounded) linear operator. By applying the stochastic target theory and the backward reachability approach, we show that the original (possibly discontinuous) value function can be represented by the zero-level set of the auxiliary (continuous) value function. The auxiliary value function is obtained from the penalized unconstrained stochastic control problem (in infinite dimensions) that includes an additional control variable as a consequence of the (infinite-dimensional) martingale representation theorem. We then prove that the auxiliary value function is a unique (continuous) viscosity solution to the associated Hamilton–Jacobi–Bellman (HJB) equation in infinite dimensions. Note that the viscosity analysis developed in our paper generalizes that presented in the existing literature, since the corresponding infinite-dimensional HJB equation includes an additional operator-valued control variable in the Hamiltonian maximization and depends on an additional initial state variable.

Embedding of a truncated vector lattice into its universal completion

We prove in a purely algebraic way that if L is an Archimedean truncated vector lattice then there exists a positive element e in the universal completion Lu of L such that the truncation of L is provided by meet with e. Previous representations of truncated vector lattices by almost-finite extended-real continuous valued functions can be obtained as consequences.

The Gibbs–Wilbraham phenomenon in the approximation of [formula omitted] by using Lagrange interpolation on the Chebyshev–Lobatto nodal systems

Along this study we find and deeply analyze a new Gibbs phenomenon. As far as we know, this type of behavior, in different contexts, is connected with functions having jump discontinuities. In our case it is related to the behavior of the Lagrange interpolation polynomials of the continuous absolute value function. Our study is related to the error of the Lagrange polynomial interpolants of the function |x| on [−1,1] taking as nodal system the m+2 nodes of the extended Chebyshev polynomial of the second kind, obtaining that the error behaves like a function of order O(1/m). A detailed description and approximation of the function is presented.

State-Continuity Approximation of Markov Decision Processes via Finite Element Methods for Autonomous System Planning

Motion planning under uncertainty for an autonomous system can be formulated as a Markov Decision Process with a continuous state space. In this paper, we propose a novel solution to this decision-theoretic planning problem that directly obtains the continuous value function with only the first and second moments of the transition probabilities, alleviating the requirement for an explicit transition model in the literature. We achieve this by expressing the value function as a linear combination of basis functions and approximating the Bellman equation by a partial differential equation, where the value function can be naturally constructed using a finite element method. We have validated our approach via extensive simulations, and the evaluations reveal that to baseline methods, our solution leads to in terms of path smoothness, travel distance, and time costs.

Workforce Scheduling in the Era of Crowdsourced Delivery

Using crowdsourced delivery capacity, that is, individuals offering their vehicle and their time to perform deliveries, can allow companies to provide faster delivery options and more easily accommodate fluctuations in demand. However, because of the uncertainty associated with crowdsourced delivery capacity, ensuring service quality is more challenging. To prevent or mitigate any negative effects of the uncertainty associated with crowdsourced delivery capacity, companies may choose to also have a scheduled delivery workforce that they can control more effectively. We investigate continuous approximation and value function approximation methods for scheduling this workforce, that is, deciding their shifts (start time and duration) to achieve a service level target at minimum cost. An extensive computational study demonstrates the efficacy of our methods and provides insights into the use of crowdsourced delivery capacity.

Piecewise Affine Feedback Control for Approximate Solution of the Target Control Problem

This paper presents an approach for approximate solution of the target control problem for a class of nonlinear systems on a finite time interval. The main idea is to use a comparison principle from dynamic programming theory together with a special class of piecewise affine value functions for the piecewise linearized form of the original system of ODEs. Two cases are considered: with continuous value function and with discontinuous one. The resulting feedback control functions are also piecewise affine, continuous or discontinuous respectively. For both cases the theorems on sufficient conditions for solving the feedback control problem are formulated and proved. The appropriate computational procedures complement theoretical results.

Uniform Integrability for Fuzzy Random Variables

In this paper the uniform integrability of fuzzy random variable is introduced. An attempt is made to study the uniformly absolutely continuous fuzzy valued functions and an equivalent condition for uniform integrable and uniformly absolutely continuous is determined.

Existence of Generalized Recursive Equilibrium in Krusell and Smith (1998)

In this paper, I define and show the existence of a generalized recursive equilibrium in Krusell and Smith (1998)’s economy with both aggregate and idiosyncratic shocks. The method to establish existence using finite horizon approximation applies equally well to economies with a finite number of agents or with a continuum of agents. With a continuum of agents, my proof relies on the compactness of the space of probability measures over a bounded interval endowed with weak topology and of the space of Lipschitz continuous value functions and weakly increasing policy functions. These properties enable the use of the Kakutani-Glicksberg-Fan Fixed Point Theorem for infinite dimensional spaces. The existence proof suggests a numerical algorithm to compute a global nonlinear recursive equilibrium. I implement the algorithm and analyze its numerical solutions for two-agent economies.

Universal Nash Equilibrium Strategies for Differential Games

The paper is concerned with a two-player nonzero-sum differential game in the case when players are informed about the current position. We consider the game in control with guide strategies first proposed by Krasovskii and Subbotin. The construction of universal strategies is given both for the case of continuous and discontinuous value functions. The existence of a discontinuous value function is established. The continuous value function does not exist in the general case. In addition, we show the example of smooth value function not being a solution of the system of the Hamilton–Jacobi equation.

A NOTE ON THE INTEGRAL REPRESENTATION OF THE VALUE FUNCTION

In this paper we propose an alternative assumption for the integral representation of the value function of Milgrom and Segal (2002). Instead of requiring that utility function has derivative almost everywhere, we impose that it has derivative in all its domain. The idea is to obtain conditions in order to apply the Lebesgue Theorem which provides at the same time an absolutely continuous value function and its integral representation. Our assumption is technically stronger than that of Milgrom and Segal (2002) but we argue that there is a substantial gain of economic interpretation in adding it. While it is difficult to interpret absolute continuity in terms of agent's preferences, the existence of the derivative everywhere means that all agent's choices are smooth.

Increasing the Probability of Developing Affordable Systems by Maximizing and Adapting the Solution Space

The present research suggests that the size of the solution space, which for a given set of stakeholder needs is delimited by system requirements, relates to the probability of finding affordable solutions. As a result, the effectiveness of tradespace exploration techniques is limited by its size and internal ordering. Therefore, we suggest that there exist models to elicit and use requirements that, for a given set older needs, could facilitate the maximization of the solutionof stakehspace so that the probability of finding more affordable solutions during tradespace exploration is also maximized.The present research proposes a mathematical model of the requirements elicitation process that facilitates defining performance objectives for the requirements elicitation process in striving for system affordability. The uniqueness of this research lays on two elements. First, the requirements elicitation process is mathematically modeled so that their objectives with respect to the effects on the solution space can be mathematically, and thus rigorously, described. Second, the system of interest focuses on the definition of the solutionspace as a driver for system affordability instead of on its actual exploration. The present research closes therefore the loop between stakeholder needs, system requirements, solutionspaces, and system affordability. The results of the present research are generalized to discrete requirements, fuzzy requirements, and continuous requirements or value functions.

In This Issue

In This Issue

A review of stochastic algorithms with continuous value function approximation and some new approximate policy iteration algorithms for multidimensional continuous applications

We review the literature on approximate dynamic programming, with the goal of better understanding the theory behind practical algorithms for solving dynamic programs with continuous and vector-valued states and actions and complex information processes. We build on the literature that has addressed the well-known problem of multidimensional (and possibly continuous) states, and the extensive literature on model-free dynamic programming, which also assumes that the expectation in Bellman’s equation cannot be computed. However, we point out complications that arise when the actions/controls are vector-valued and possibly continuous. We then describe some recent research by the authors on approximate policy iteration algorithms that offer convergence guarantees (with technical assumptions) for both parametric and nonparametric architectures for the value function.

Stationary monetary equilibria with strictly increasing value functions and non-discrete money holdings distributions: An indeterminacy result

Abstract In this paper, we present a search model with divisible money in which there exists a continuum of monetary equilibria with strictly increasing continuous value functions and with non-discrete money holdings distributions.

In search of the essential binary discrete particle swarm

The particle swarm optimization algorithm is an innovative and competitive optimization technique in evolutionary computation. It has been found to be extremely effective in solving a wide range of problems with real-parameter representation; however, it is of low efficiency in dealing with the discrete problems. In this paper, the particle swarm algorithm is broken down into its essential components, and alternative interpretations of those components are proposed. It is simpler and more powerful than the algorithms available. Experimental results show that this algorithm is faster than the standard binary discrete PSO on two suites of test functions, and that accuracy is improved for most benchmark functions used. One suite concerns about binary encoding problems, the other is about continuous-valued functions. A queen informant is also introduced. It does not increase the number of function evaluations; however, it appears it greatly speeds up the convergence.

On structure of the value function

AbstractThe value function (or the optimal result function) arising in optimal control problems with the Bolza pay-off functionals is studied as the unique minimax or viscosity solution of a corresponding boundary problem for the Hamilton-Jacobi-Bellman (the dynamic programming equiation) equation. It is obtained for one-dimensional state space, that the continuous value function is not differentiable on at the most countable set of the Rankine-Hugoniot lines. The structure of the set of singular points is important to constructions of optimal synthesis in optimal control problems.

Proper scoring rules with arbitrary value functions

Abstract A scoring rule is proper if it elicits an expert’s true beliefs as a probabilistic forecast, and it is strictly proper if it uniquely elicits an expert’s true beliefs. The value function associated with a (strictly) proper scoring rule is (strictly) convex on any convex set of beliefs. This paper gives conditions on compact sets of possible beliefs Θ that guarantee that every continuous value function on Θ is the value function associated with some strictly proper scoring rule. Compact subsets of many parametrized sets of distributions on R k satisfy these conditions.

Value Iteration for (Switched) Homogeneous Systems

In this note, we prove that dynamic programming value iteration converges uniformly for discrete-time homogeneous systems and continuous-time switched homogeneous systems. For discrete-time homogeneous systems, rather than discounting the cost function (which exponentially decreases the weights of the cost of future actions), we show that such systems satisfy approximate dynamic programming conditions recently developed by Rantzer, which provides a uniform bound on the convergence rate of value iteration over a compact set. For continuous-time switched homogeneous system, we present a transformation that generates an equivalent discrete-time homogeneous system with an additional ldquosamplingrdquo input for which discrete-time value iteration is compatible, and we further show that the inclusion of homogeneous switching costs results in a continuous value function.