Bellman Optimality Research Articles

Optimal Model-Free Mean-Square Consensus for Multi-Agents with Markov Switching Topology

Due to the real applications, optimal consensus reinforcement learning with switching topology is still challenging due to the complexity of topological changes. This paper investigates the optimal consensus control problem for discrete multi-agent systems under Markov switching topologies. The goal is to design an appropriate algorithm to find the optimal control policies that minimize the performance index while achieving consensus among the agents. The concept of mean-square consensus is introduced, and the relationship between consensus error and tracking error to achieve mean-square consensus is studied. A performance function for each agent under switching topologies is established and a policy iteration algorithm using system data is proposed based on the Bellman optimality principle. The theoretical analysis shows that the consensus error realizes mean-square consensus and the performance function is optimized. The efficacy of the suggested approach is confirmed by numerical simulation using an actor–critic neural network. As a result, the value function is the optimum and the mean-square consensus can be reached using this technique.

Dual-timescale resource management for multi-type caching placement and multi-user computation offloading in Internet of Vehicle

In Internet of Vehicle (IoV), edge computing can effectively reduce task processing delays and meet the real-time needs of connected-vehicle applications. However, since the requirements for caching and computing resources vary across heterogeneous vehicle requests, a new challenge is posed on the resource management in the three-tier cloud–edge–end architecture, particularly when multi users offload tasks in the same time. Our work comprehensively considers various scenarios involving the deployment of multiple caching types from multi-users and the distinct time scales of offloading and updating, then builds a joint optimization caching placement, computation offloading and computational resource allocation model, aiming to minimize overall latency. Meanwhile, to better solving the model, we propose the Multi-node Collaborative Caching, Offloading, and Resource Allocation Algorithm (MCCO-RAA). MCCO-RAA utilizes dual time scales to optimize the problem: employing a Bellman optimization idea-based multi-node collaborative greedy caching placement strategy at large time scales, and a computational offloading and resource allocation strategy based on a two-tier iterative Deep Deterministic Policy Gradient (DDPG) and cooperative game at small time scales. Experimental results demonstrate that our proposed scheme achieves a 28% reduction in overall system latency compared to the baseline scheme, with smoother latency variations under different parameters.

Optimized Backstepping Cooperative Control for Output-Constrained Stochastic Nonlinear Network Systems via a Multibridge-Hole Function.

In this article, a new leader-following tracking control approach is investigated for stochastic multiagent systems with multibridge-hole output constraints. The multibridge-hole output constraints mean that the output of the system is constrained in some intervals and unconstrained in other intervals. The constrained and unconstrained intervals can be set arbitrarily. By designing a new shift function to construct the barrier Lyapunov function, the optimal controller is constructed by combining the backstepping technique with the adaptive dynamic programming technique. The model network is used to estimate the unknown disturbances and uncertainty terms in the system. The critic network and the actor network are constructed such that the designed controller adheres to the Bellman optimality principle and gives the optimal solution of the system. The proposed control method is versatile and compatible with various types of output constrained control problems, such as unconstrained control problems, constrained control problems, and delay constrained problems without changing the structure of the controller. Finally, some simulation results are given to verify the effectiveness of the method.

Research and application of the flatness target curve discrete dynamic programming based on two-dimensional decision making

The flatness target curve (FTC) is a fundamental process model in cold rolling flatness control, which directly affects the quality control effect of the strip. The accurate and efficient setting of the flatness target curve is the key to ensuring continuous rolling and is a common problem faced by various production companies. This paper establishes a discrete dynamic planning (DDP) setting method for FTC with two-dimensional decision-making. To quantitatively analyze the roles of the coefficients in the FTC, the preprocessing process is established using standardized processing methods and statistical analysis methods. The flatness target curve discrete dynamic programming (FTC-DDP) is based on Bellman optimization theory with a dual structure of local and system decision-making. The decision-making functions are established based on the effect of each coefficient on FTC geometric characteristics. The concepts of “curvature factor” and “wedge factor” are innovatively proposed to obtain the decision-making value functions. The regression analysis establishes the solution equations of even number coefficients, odd number coefficients, and gain coefficients, respectively. Roll bending and tilting roll analyze the FTC’s ability to correct flatness defects. Meanwhile, utilizing a 1450 mm five-stand six-roll cold rolling mill, it is verified that FTC-DDP can quickly and accurately set the FTC, which provides an effective solution for obtaining high-precision and high-quality cold-rolled strips. Application results show that FTC-DDP can reduce the difficulty of setting up FTC by more than 70 %, reduce the setup time to milliseconds, and improve the setup accuracy by more than 50 %.

A review of dynamic optimization in aquaculture production economics

AbstractDynamic optimization (DO) has been applied to aquaculture industry to determine optimal management strategies of aquaculture production systems. Optimal control of feeding, optimal stocking successions, and harvesting time among other factors has been studied. This work shows a review of the application of optimal control theory (OCT) and dynamic programming in economics and management of aquaculture. The Pontriagyn's maximum principle and the Bellman's optimality are the most commonly used mechanisms to solve control problems that optimize the producer's benefit or cost. Recently, model‐based predictive control has also been applied. Such tools have been used in aspects involving the design, planning, and monitoring of variables relevant to the optimal management of the culture system. It was found that in aquaculture there is low scientific productivity in the application of DO in bioeconomic models. In this review, an example of OCT applied to the control of feeding of farmed tilapia using a bioeconomic model is shown. DO represents a useful tool for optimal decision‐making and this review discusses the implications of the use of DO in aquaculture and recommendations for its use in the future.

Differential graphical game‐based multi‐agent tracking control using integral reinforcement learning

AbstractThis paper studies the cooperative tracking control problem of interacted multi‐agent systems (MASs) under undirected communication. Based on differential graphical game theory, the MAS tracking control problem is formulated as an infinite horizon cooperative differential graphical game‐theoretic tracking control framework, where a multi‐objective optimization problem is designed and then cast into a Pareto‐equivalent single‐objective optimization problem using a scalarization method. Necessary and sufficient conditions for the existence of the Pareto‐optimal strategy to the game theoretic tracking control are established, where it has been proven that the solution to the integral Bellman optimality equation leads to Pareto‐optimal strategy. Then, an off‐policy integral reinforcement learning scheme to find optimal control strategy using a pure data‐driven manner is developed, which consumes less computation efforts than the traditional learning scheme. Simulated results are conducted to validate the effectiveness of the proposed game and IRL‐based tracking control method.

Optimal control for both forward and backward discrete-time systems

Forward discrete-time systems use past information to update the current state, while backward discrete-time systems use future information to update the current state. This study focuses on optimal control problems within the context of forward and backward discrete-time systems. We begin by investigating a general optimal control problem for both forward and backward discrete-time systems. Leveraging the inherent properties of these systems and the Bellman optimality principle, we derive recursive equations as a means to solve such optimal control problems. Using these recursive equations, we obtain analytical expressions for both the optimal controls and optimal values of bang–bang and linear quadratic optimal control problems. Finally, we present a numerical example and an industrial wastewater treatment problem to illustrate and demonstrate our findings.

Boundary Optimal Control for Parabolic Distributed Parameter Systems With Value Iteration.

A reinforcement learning-based boundary optimal control algorithm for parabolic distributed parameter systems is developed in this article. First, a spatial Riccati-like equation and an integral optimal controller are derived in infinite-time horizon based on the principle of the variational method, which avoids the complex semigroups and operator theories. Using state data along the system trajectory, a value iteration algorithm via the Bellman optimality principle is proposed to obtain the solution of the spatial Riccati-like equation and the optimal control law. The convergence of the value iteration algorithm is proved. Subsequently, an approximation scheme based on weighted residuals is developed to implement the value iteration algorithm, where radial basis functions are chosen as the basic functions to approximate the solution of the spatial Riccati-like equation. Simulations on the diffusion-reaction process demonstrate the effectiveness of the developed method.

Barrier Lyapunov Function-Based Safe Reinforcement Learning for Autonomous Vehicles With Optimized Backstepping.

Guaranteed safety and performance under various circumstances remain technically critical and practically challenging for the wide deployment of autonomous vehicles. Safety-critical systems in general, require safe performance even during the reinforcement learning (RL) period. To address this issue, a Barrier Lyapunov Function-based safe RL (BLF-SRL) algorithm is proposed here for the formulated nonlinear system in strict-feedback form. This approach appropriately arranges and incorporates the BLF items into the optimized backstepping control method to constrain the state-variables in the designed safety region during learning. Wherein, thus, the optimal virtual/actual control in every backstepping subsystem is decomposed with BLF items and also with an adaptive uncertain item to be learned, which achieves safe exploration during the learning process. Then, the principle of Bellman optimality of continuous-time Hamilton-Jacobi-Bellman equation in every backstepping subsystem is satisfied with independently approximated actor and critic under the framework of actor-critic through the designed iterative updating. Eventually, the overall system control is optimized with the proposed BLF-SRL method. It is furthermore noteworthy that the variance of the attained control performance under uncertainty is also reduced with the proposed method. The effectiveness of the proposed method is verified with two motion control problems for autonomous vehicles through appropriate comparison simulations.

Kernel-based diffusion approximated Markov decision processes for autonomous navigation and control on unstructured terrains

We propose a diffusion approximation method to the continuous-state Markov decision processes that can be utilized to address autonomous navigation and control in unstructured off-road environments. In contrast to most decision-theoretic planning frameworks that assume fully known state transition models, we design a method that eliminates such a strong assumption that is often extremely difficult to engineer in reality. We first take the second-order Taylor expansion of the value function. The Bellman optimality equation is then approximated by a partial differential equation, which only relies on the first and second moments of the transition model. By combining the kernel representation of the value function, we design an efficient policy iteration algorithm whose policy evaluation step can be represented as a linear system of equations characterized by a finite set of supporting states. We first validate the proposed method through extensive simulations in 2D obstacle avoidance and 2.5D terrain navigation problems. The results show that the proposed approach leads to a much superior performance over several baselines. We then develop a system that integrates our decision-making framework with onboard perception and conduct real-world experiments in both cluttered indoor and unstructured outdoor environments. The results from the physical systems further demonstrate the applicability of our method in challenging real-world environments.

Multiagent, Multitarget Path Planning in Markov Decision Processes

Missions for autonomous systems often require agents to visit multiple targets in complex operating conditions. This work considers the problem of visiting a set of targets in minimum time by a team of non-communicating agents in a Markov decision process (MDP). The single-agent problem is at least NP-complete by reducing it to a Hamiltonian path problem. We first discuss an optimal algorithm based on Bellman's optimality equation that is exponential in the number of target states. Then, we trade-off optimality for time complexity by presenting a suboptimal algorithm that is polynomial at each time step. We prove that the proposed algorithm generates optimal policies for certain classes of MDPs. Extending our procedure to the multi-agent case, we propose a target partitioning algorithm that approximately minimizes the expected time to visit the targets. We prove that our algorithm generates optimal partitions for clustered target scenarios. We present the performance of our algorithms on random MDPs and gridworld environments inspired by ocean dynamics. We show that our algorithms are much faster than the optimal procedure and more optimal than the currently available heuristic.

Real-Time Leak Location of Long-Distance Pipeline Using Adaptive Dynamic Programming.

In traditional leak location methods, the position of the leak point is located through the time difference of pressure change points of both ends of the pipeline. The inaccurate estimation of pressure change points leads to the wrong leak location result. To address it, adaptive dynamic programming is proposed to solve the pipeline leak location problem in this article. First, a pipeline model is proposed to describe the pressure change along pipeline, which is utilized to reflect the iterative situation of the logarithmic form of pressure change. Then, under the Bellman optimality principle, a value iteration (VI) scheme is proposed to provide the optimal sequence of the nominal parameter and obtain the pipeline leak point. Furthermore, neural networks are built as the VI scheme structure to ensure the iterative performance of the proposed method. By transforming into the dynamic optimization problem, the proposed method adopts the estimation of the logarithmic form of pressure changes of both ends of the pipeline to locate the leak point, which avoids the wrong results caused by unclear pressure change points. Thus, it could be applied for real-time leak location of long-distance pipeline. Finally, the experiment cases are given to illustrate the effectiveness of the proposed method.

Traffic efficiency and fairness optimisation for autonomous intersection management based on reinforcement learning

Autonomous Intersection Management (AIM) for high-level Connected and Automated Vehicles (CAVs) has evolved from rule-based to optimisation-based policies. However, at congested major-minor intersections, optimising solely for efficiency can negatively impact vehicle fairness. This study addresses this issue by proposing a deep reinforcement learning approach that optimises both traffic efficiency and fairness for AIM. In the modelled multi-objective Markov decision process, traffic fairness is measured by the difference between the crossing order and the approaching order of CAVs, while traffic efficiency is measured by average travel time. With unknown preferences of the objectives, Bellman optimality equation is generalised to obtain the optimal policies over the space of all possible preferences during the iterative training process. The effectiveness of the proposed method is evaluated in a simulated real-world intersection and compared with three benchmark policies, including the fairest policy for AIM: first-come-first-served. The learned policies perform best in reducing overall average vehicle delay, and demonstrate outstanding performance in balancing traffic fairness and efficiency.

Learning-Based Event-Triggered Tracking Control for Nonlinear Networked Control Systems With Unmatched Disturbance

This article concentrates on optimal tracking control for a class of nonlinear networked systems subjecting to limited network bandwidth and unmatched disturbance. Given the models of the control and reference systems, the considered optimal tracking control issue is initially formulated as a minimax optimization problem. Then, with the introduction of an event-triggered mechanism used for saving bandwidth, the formulated problem is transformed into solving an event-based Hamilton-Jacobi-Isaacs (HJI) equation by recurring to the Bellman optimality theory. Based on the HJI equation, we demonstrate that the stability of the concerned system in the sense of uniformly ultimately bounded (UUB) can be guaranteed with the envisioned optimal control and worst disturbance policies. Here, the disturbance policy can be varied periodically while the control policy can only be updated at event-triggering instants, which differs from the existed researches. Furthermore, we propose a reinforcement learning (RL)-based algorithm to handle the constructed HJI equation and thus settle the studied tracking control problem. The effectiveness of the algorithm is finally validated by both theoretical analysis and simulations.

Me, myself and I: A general theory of non-Markovian time-inconsistent stochastic control for sophisticated agents

We develop a theory for continuous-time non-Markovian stochastic control problems which are inherently time-inconsistent. Their distinguishing feature is that the classical Bellman optimality principle no longer holds. Our formulation is cast within the framework of a controlled non-Markovian forward stochastic differential equation, and a general objective functional setting. We adopt a game-theoretic approach to study such problems, meaning that we seek for sub-game perfect Nash equilibrium points. As a first novelty of this work, we introduce and motivate a refinement of the definition of equilibrium that allows us to establish a direct and rigorous proof of an extended dynamic programming principle, in the same spirit as in the classical theory. This in turn allows us to introduce a system consisting of an infinite family of backward stochastic differential equations analogous to the classical HJB equation. We prove that this system is fundamental, in the sense that its well-posedness is both necessary and sufficient to characterise the value function and equilibria. As a final step we provide an existence and uniqueness result. Some examples and extensions of our results are also presented.

A Minimum Principle for Stochastic Optimal Control Problem with Interval Cost Function

In this paper, we study an optimal control problem in which their cost function is interval-valued. Also, a stochastic differential equation governs their state space. Moreover, we introduce a generalized version of Bellman's optimality principle for the stochastic system with an interval-valued cost function. Also, we obtain the Hamilton–Jacobi–Bellman equations and their control decisions. Two numerical examples happen in finance in which their cost function are interval-valued functions, illustrating the efficiency of the discussed results. The obtained results provide significantly reliable decisions compared to the case where the conventional cost function is applied.

Features in solving individual tasks to develop service-oriented networks using dynamic programming

The object of this study is an approach to solving the problems of designing service-oriented networks that warn about emergencies using dynamic programming. The main issue is the complexity of algorithmization of processes that describe the achievement of an optimal solution in multi-stage nonlinear problems. The possibilities of applying the Bellman optimality principle for solving the set tasks for the purpose of their application in the field of engineering and technology are determined. Based on the Bellman functional equation, a model of the optimal number of sensors in the monitoring system for warning of emergencies was built. A feature of the design is that using the classical Bellman equation, it is proposed to solve problems of various technical directions, provided that the resource determines what exactly makes it possible to optimize work in any way. Important with this approach is the planning of the action as an element of some problem with the augmented state. After that, the proposed structure in formal form extends to other objects. A problem was proposed and considered, which confirmed the mathematical calculations, as a result of which an optimal plan for replacing the sensors of the system was obtained; and the possibilities of significant cost reduction were identified. In the considered example, an optimal plan for replacing the system sensors was compiled and the possibility of reducing costs by 31.9 % was proved. The proposed option was used in the development of information technology for modeling a service-oriented network based on energy-efficient long-range protocols; some of the identified features were further developed in the design of a recommendation system for issuing loans and developing an interactive personnel training system

Integer optimal control problems with total variation regularization: Optimality conditions and fast solution of subproblems

We investigate local optimality conditions of first and second order for integer optimal control problems with total variation regularization via a finite-dimensional switching-point problem. We show the equivalence of local optimality for both problems, which will be used to derive conditions concerning the switching points of the control function. A non-local optimality condition treating back-and-forth switches will be formulated. For the numerical solution, we propose a proximal-gradient method. The emerging discretized subproblems will be solved by employing Bellman’s optimality principle, leading to an algorithm which is polynomial in the mesh size and in the admissible control levels. An adaption of this algorithm can be used to handle subproblems of the trust-region method proposed in Leyffer and Manns, ESAIM: Control Optim. Calc. Var. 28 (2022) 66. Finally, we demonstrate computational results.

Reinforcement Learning With Non-Cumulative Objective

In reinforcement learning, the objective is almost always defined as a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">cumulative</i> function over the rewards along the process. However, there are many optimal control and reinforcement learning problems in various application fields, especially in communications and networking, where the objectives are not naturally expressed as summations of the rewards. In this paper, we recognize the prevalence of non-cumulative objectives in various problems, and propose a modification to existing algorithms for optimizing such objectives. Specifically, we dive into the fundamental building block for many optimal control and reinforcement learning algorithms: the Bellman optimality equation. To optimize a non-cumulative objective, we replace the original <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">summation operation</i> in the Bellman update rule with a generalized operation corresponding to the objective. Furthermore, we provide sufficient conditions on the form of the generalized operation as well as assumptions on the Markov decision process under which the globally optimal convergence of the generalized Bellman updates can be guaranteed. We demonstrate the idea experimentally with the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">bottleneck</i> objective, i.e., the objectives determined by the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">minimum</i> reward along the process, on classical optimal control and reinforcement learning tasks, as well as on two network routing problems on maximizing the flow rates.

Optimal Control and Adaptive Learning for Stabilization of a Quadrotor-type Unmanned Aerial Vehicle via Approximate Dynamic Programming

The development of an optimal controller for stabilization of a quadrotor system using an adaptive critic structure based on policy iteration schemes is proposed in this paper. This approach is inserted in the context of Approximate Dynamic Programming and it is used to solve optimal decision problems on-line, without requiring complete knowledge of the system dynamics model to be controlled. The main feature of the adaptive critic design method that allows for on-line implementation is that it solves the Bellman optimality equation in a forward-in-time fashion, whereas traditional dynamic programming requires a backward-in-time procedure. This feedback control design technique is able to tune the controller parameters on-line in the presence of variations in plant dynamics and external disturbances using data measured along the system trajectories. Computational simulation results based on a quadrotor model demonstrate the effectiveness of the proposed control scheme.