Dynamic Programming Technique Research Articles

Recursive two-stage evaluation model for dynamic decision making under ambiguity

In this paper, the two-stage-evaluation (TSE) model for decision making under ambiguity (He 2021) is extended to intertemporal setting in an axiomatic approach. The first set of axioms employed are commonly adopted for dynamic non-expected utility models in the literature. Besides these regular axioms, I also assume dynamic consistency and conditions which deliver a static TSE for consumption plans only pay non-zero consequences for one period. It is shown that these axioms hold if and only if these exists a recursively defined evaluation utility model representing decision maker (DM)’s preferences over consumption plans conditional on arriving at any node in an event tree. Such a recursive form implies that one can apply dynamic programming technique (rolling back the decision tree) to solve a dynamic decision making problem under TSE model. It can be shown that the solution for the recursively defined dynamic TSE model exists uniquely. Due to the “small domain” setup, the agent is short-sighted in the sense that they only process subjective probabilities over events defined on one period uncertainty over next period states, which differs from “far-sighted” assumption in most extant models that assume subjective probabilities exist over events defined on multiple periods. It is shown that under some extra conditions, our DM applies Bayes’ rule “updating’ her subjective beliefs.

Travel Salesman Problem Using Dynamic Programming

In the fields of operations research and computer science, the traveling salesman problem (also known as TSP) is a classic algorithmic issue. One method for solving TSP is dynamic programming, which identifies the optimal path at the lowest possible cost. It starts by determining the shortest path between two points, and then it expands to identify routes to other sites. In order to solve the Travel salesman Problem, Dynamic Programming was used in this study as a a solution technique. In addition to a real-world application from AM-EXPRESS NIGERIA LIMITED, illustrative examples of travel salesman difficulties were taken into consideration and handled utilizing the dynamic programming technique. The finding's outcome showed that the least amount of money needed to provide logistical services is ₦6500, and the shortest possible paths or routes were found.

Robust optimal control for uncertain wheeled mobile robot based on reinforcement learning: ADP approach

This paper presents a robust optimal control approach for the wheel mobile robot system, which considers the effects of external disturbances, uncertainties, and wheel slipping. The proposed method utilizes an adaptive dynamic programming (ADP) technique in conjunction with a disturbance observer. Initially, the system's state space model is formulated through the utilization of kinematic and dynamic models. Subsequently, the ADP method is employed to establish an online adaptive optimal controller, which solely relies on a single neural network for the purpose of function approximation. The utilization of the disturbance observer in conjunction with the compensation controller serves to alleviate the effects of disturbances. The Lyapunov theorem establishes the stability of the complete closed-loop system and the convergence of the weights of the neural network. The proposed approach has been shown to be effective through simulation under the effect of the disturbances and the change of the desired trajectory.

Equilibrium strategies in a defined benefit pension plan game with regime switching

This paper studies the optimal management of an aggregated overfunded pension plan of defined benefit type as a stochastic differential game in a Markovian regime-switching environment. In this asymmetric two-player game, the worker participants act collectively as a single player that claims a share of the surplus, and the sponsoring firm acts as the player who cares about the investment of the surplus fund assets. We formulate the game model in two scenarios, and use the dynamic programming technique to derive the coupled Hamilton-Jacobi-Bellman equations in each scenario. Furthermore, we propose an efficient algorithm that incorporates a kind of quasi-Newton method and the Runge-Kutta scheme to obtain the equilibrium strategies. Finally, we employ a numerical example to verify our algorithm and analyze the impact of model parameters.

Maximal size constraint community search over bipartite graphs

Community search over bipartite graphs is a crucial issue with significant applications in various domains, including group recommendations, fraud detection and representation learning. Current research primarily focuses on the structural cohesiveness constraint between two sets of vertices, often neglecting the community’s size constraint. Unfortunately, this oversight may lead to communities of sizes ranging from excessively small to substantially large, thereby affecting their applicability. Therefore, in this paper, we focus on the maximal size constraint community search over bipartite graphs, which not only takes into account the (α,β) degree constraints for each set of vertices but also ensures that the number of vertices in each set is limited to ξ and ζ, respectively. To solve this problem, we first introduce two competitive algorithms, namely Ordering-based Expansion Algorithm (OEA) and Followers-based Peeling Algorithm (FPA), wherein they greedily expand or peel vertices, respectively, to identify the maximal one in a breadth-first manner, utilizing a novel vertex ordering and followers concept. To further accelerate the computation, we propose an advanced Expand-and-Filter Algorithm (EFA++) that first acquires a slightly larger community and subsequently executes a filtering phase employing a dynamic programming technique. Furthermore, we define the “Affected Area” for edge insertion or deletion that caters for dynamic scenarios. Finally, the theoretical analysis and extensive experimental evaluation on several real-life datasets demonstrate the effectiveness and efficiency of the proposed algorithms.

Adaptive dynamic programming and distributionally robust optimal control of linear stochastic system using the Wasserstein metric

SummaryIn this paper, we consider the optimal control of unknown stochastic dynamical system for both the finite‐horizon and infinite‐horizon cases. The objective of this paper is to find an optimal controller to minimize the expected value of a function which depends on the random disturbance. Throughout this paper, it is assumed that the mean vector and covariance matrix of the disturbance distribution is unknown. An uncertainty set in the space of mean vector and the covariance matrix is introduced. For the finite‐horizon case, we derive a closed‐form expression of the unique optimal policy and the opponents policy that generates the worst‐case distribution. For the infinite‐horizon case, we simplify the Riccati equation obtained in the finite‐hozion setting to an algebraic Riccati equation, which can guarantee the existence of the solution of the Riccati equation. It is shown that the resulting optimal policies obtained in these two cases can stabilize the expected value of the system state under the worst‐case distribution. Furthermore, the unknown system matrices can also be explicitly computed using the adaptive dynamic programming technique, which can help compute the optimal control policy by solving the algebraic Riccati equation. Finally, a simulation example is presented to demonstrate the effectiveness of our theoretical results.

Optimized Backstepping Cooperative Control for Output-Constrained Stochastic Nonlinear Network Systems via a Multibridge-Hole Function.

In this article, a new leader-following tracking control approach is investigated for stochastic multiagent systems with multibridge-hole output constraints. The multibridge-hole output constraints mean that the output of the system is constrained in some intervals and unconstrained in other intervals. The constrained and unconstrained intervals can be set arbitrarily. By designing a new shift function to construct the barrier Lyapunov function, the optimal controller is constructed by combining the backstepping technique with the adaptive dynamic programming technique. The model network is used to estimate the unknown disturbances and uncertainty terms in the system. The critic network and the actor network are constructed such that the designed controller adheres to the Bellman optimality principle and gives the optimal solution of the system. The proposed control method is versatile and compatible with various types of output constrained control problems, such as unconstrained control problems, constrained control problems, and delay constrained problems without changing the structure of the controller. Finally, some simulation results are given to verify the effectiveness of the method.

Adaptive fault-tolerant optimal control for hypersonic vehicles with state constrains based on adaptive dynamic programming

This paper addresses a novel adaptive fault-tolerant control (FTC) design based on adaptive dynamic programming (ADP) technique for hypersonic vehicle (HSV) subject to actuator fault and state constrains. The total control input is constructed by the combination of backstepping-based adaptive FTC and the ADP-based adaptive optimal control to improve the tracking performance and fault-tolerant capacity. By introducing a barrier Lyapunov function (BLF) to deal with the state constraints, a backstepping-based FTC scheme is designed to transform the tracking control problem into an equivalent optimal control problem. Subsequently, an adaptive optimal control strategy is developed by using the ADP technique to provide a supplementary control action. A critic network is constructed to solve the Hamilton–Jacobi–Bellman (HJB) equation online. The convergence properties of the closed-loop system are developed by utilizing the Lyapunov stability theory. Finally, simulation results are carried out to illustrate the effectiveness and efficiency of the proposed adaptive ADP-based FTC strategy.

Optimal captured power control of variable speed wind turbine systems: Adaptive dynamic programming approach

SummaryAn adaptive optimal controller is proposed in this paper to maximize the captured power of a variable speed wind power system. The proposed controller is a combination of optimal and adaptive control components. The Adaptive Dynamic Programming technique is used to design the optimal control component to overcome the nonlinear problem of system dynamics and ensure stability. While the neural network is used to approximate unknown disturbances and system uncertainties. After that, the adaptive control component fully compensates for the effects of these unknown elements. Neither optimal nor adaptive control components necessitate prior knowledge of system dynamics. Furthermore, the approximation network updates only the weight matrix norm rather than the weight matrix of the neural network in each interval time, which significantly reduces computation. The stability analysis of the closed‐loop system is obtained using Lyapunov stability theory. The correctness and robustness of the control scheme are validated in two different scenarios using MATLAB/Simulink. The presented robust adaptive optimal controller is also compared to other existing controllers to demonstrate its benefits.

Reinforcement Learning for Dual-Control Aircraft Six-Degree-of-Freedom Attitude Control with System Uncertainty

This article proposes a near-optimal control strategy based on reinforcement learning, which is applied to the six-degree-of-freedom (6-DoF) attitude control of dual-control aircraft. In order to solve the problem that the existing reinforcement learning is difficult to apply to the high-dimensional multiple-input multiple-output (MIMO) systems, the Long Short-Term Memory (LSTM) neural network is introduced to replace the polynomial network in the adaptive dynamic programming (ADP) technique. Meanwhile, based on the Lyapunov method, a novel online adaptive updating law of LSTM neural network weights is given, and the stability of the system is verified. In the simulation process, the algorithm proposed in this article is applied to the six-degree-of-freedom attitude control problem of dual-control aircraft with system uncertainty. The simulation results show that the algorithm can achieve near-optimal control.

An optimal investment consumption model for retirees with no health insurance

Retirees meet a number of problems as they are growing older which needs persistent attention. Hence, without a doubt, the outcomes of the financial markets influence the choices that people make when nearing retirement. In our model, the stock price dynamics follow Geometric Brownian motion (GBM) and our goal was to optimize the expected discounted utility of consumption and terminal wealth whilst considering health expenses. The investment return process comprises risk free asset and risky assets, and the health expenses. We choose power utility functions where comprehensive solutions for Hyperbolic Absolute Risk Aversion (HARA) utility functions are obtained and optimal investment, consumption and health expenditure strategies are derived by applying dynamic programming and variable change technique on the Hamilton-Jacobi-Bellman (HJB) equations. In our numerical results it showed various effects of some economic and market parameters on the optimal investment, consumption and health expense strategies. The inflation price market risk governs the amount invested in stock, bond and also how much to be put in health to sustain a given period of the retiree's lifetime. As the health welfare rate R increases, the proportion of wealth invested in the stock increases. We also investigated the effects of the high correlation coefficients and low correlation coefficients on consumption and income rate respectively. As the constant variance discounting coefficient increases, seasoned enterprise annuity retirees decrease their allocation to the risky assets. Finally, a numerical example is presented to depict the effects of financial parameters on the optimal investment strategy with health expenditure.

Approximate optimal and safe coordination of nonlinear second-order multirobot systems with model uncertainties

This paper investigates the approximate optimal coordination for nonlinear uncertain second-order multi-robot systems with guaranteed safety (collision avoidance) Through constructing novel local error signals, the collision-free control objective is formulated into an coordination optimization problem for nominal multi-robot systems. Based on approximate dynamic programming technique, the optimal value functions and control policies are learned by simplified critic-only neural networks (NNs). Then, the approximated optimal controllers are redesigned using adaptive law to handle the effects of robots’ uncertain dynamics. It is shown that the NN weights estimation errors are uniformly ultimately bounded under proper conditions, and safe coordination of multiple robots can be achieved regardless of model uncertainties. Numerical simulations finally illustrate the effectiveness of the proposed controller.

Age of information minimization in hybrid cognitive radio networks under a timely throughput constraint

Exchanging time-critical information is prevalent in various industrial applications where low latency and timely delivery are paramount. Through this work, we consider a cognitive radio network comprised of multiple secondary users with time-sensitive traffic, and they can access the licensed channel under the hybrid interweave/underlay scheme to enhance spectrum utilization. Traffic in the secondary system is divided into two distinct categories: deadline-constrained data and status updates. Quality of service of data with expiration time, such as multimedia streams, is assessed through the timely throughput metric. However, the age of information metric is used to characterize the freshness of the status update packets, which is vital in several emerging applications. Within an interference constraint imposed by the primary user, a dynamic scheduling policy is proposed to optimize the weighted sum of the average age of information of the status update users under a strict timely throughput requirement for each user with deadline-constrained traffic. We formulate the optimization problem as a constrained Markov decision process. Then, through the drift-plus-penalty method, the problem is reduced into a series of unconstrained Markov decision problems. Finally, each subproblem is tackled using the backward dynamic programming technique. Simulation results illustrate the effect of the main system parameters, such as the PU transmitted power and transmission rate level, on the performance of the secondary system. Moreover, the model feasibility regarding the fulfillment of the constraints against PU activity is experimentally investigated under the proposed hybrid mode and classical interweave mode. The performance of the proposed policy is compared to two other low-complexity scheduling schemes, which ensure the satisfaction of the constraints; results show the performance superiority of our proposed policy.

Dissipativity in Infinite Horizon Optimal Control and Dynamic Programming

In this paper we extend dynamic programming techniques to the study of discrete-time infinite horizon optimal control problems on compact control invariant sets with state-independent best asymptotic average cost. To this end we analyse the interplay of dissipativity and optimal control, and propose novel recursive approaches for the solution of so called shifted Bellman Equations.

Characterization of a Cournot–Nash Equilibrium for a Fishery Model with Fuzzy Utilities

The article deals with the extensions of discrete-time games with infinite time horizon and their application in a fuzzy context to fishery models. The criteria for these games are the total discounted utility and the average utility in a fishing problem. However, in the fuzzy case, game theory is not the best way to represent a real fishing problem because players do not always have enough information to accurately estimate their utility in the context of fishing. For this reason, in this paper, trapezoidal-type fuzzy utility values are considered for a fishing model, and the terms of the Nash equilibrium are given in the fuzzy context, i.e., this equilibrium is represented using the partial order of the α-cuts of the fuzzy numbers; to the best of the authors’ knowledge, there is no work with this type of treatment. To obtain each equilibrium, a suitable fully determined fuzzy game is used in combination with the dynamic programming technique applied to this game in the context of fishing. The main results are (i) the Nash equilibria of the fuzzy games coincide with the Nash equilibria of the nonfuzzy games and are explicitly determined in a fishery model and (ii) the values of the fuzzy games are of trapezoidal type and are also explicitly given in the fishery model.

Efficient Search and Detection of Faint Moving Objects in Image Data

The search and detection of faint moving objects in image data can enable discoveries of small solar system bodies. To detect objects fainter than the single-frame sensitivity limit, track-before-detect methods can improve the signal-to-noise ratio of the object of interest by incoherently adding the object’s signal across multiple frames. However, traditional track-before-detect techniques can become computationally intensive over large search volumes. In this work, we present a computational approach to significantly speed up the search process by applying dynamic-programming techniques to implement the discrete X-ray transform. In this approach, image frames are processed in stages, in each of which pairs of frames are shifted and added to generate short-track segments, which are combined in later stages to form longer tracks. The algorithm speedup comes from the fact that a single short track segment can be reused multiple times for different longer tracks without the need for recomputing. Benchmark testing with simulated data shows that the method presented in this paper results in a significant reduction in runtime in comparison to a traditional track-before-detect approach. As a proof of concept, we demonstrated the applicability of the technique in performing a blind search for faint asteroids in image data collected from the Transiting Exoplanet Survey Satellite, leading to the detection of more than a thousand asteroids below the single-frame detection limit with moderate computational resources. The approach presented in this work has the potential to enable efficient discovery of previously undetected faint solar system objects across multiple orbit classes.

Optimal Model Partitioning with Low-Overhead Profiling on the PIM-based Platform for Deep Learning Inference

Recently Processing-in-Memory (PIM) has become a promising solution to achieve energy-efficient computation in data-intensive applications by placing computation near or inside the memory. In most Deep Learning (DL) frameworks, a user manually partitions a model’s computational graph (CG) onto the computing devices by considering the devices’ capability and the data transfer. The Deep Neural Network (DNN) models become increasingly complex for improving accuracy; thus, it is exceptionally challenging to partition the execution to achieve the best performance, especially on a PIM-based platform requiring frequent offloading of large amounts of data. This article proposes two novel algorithms for DL inference to resolve the challenge: low-overhead profiling and optimal model partitioning. First, we reconstruct CG by considering the devices’ capability to represent all the possible scheduling paths. Second, we develop a profiling algorithm to find the required minimum profiling paths to measure all the node and edge costs of the reconstructed CG. Finally, we devise the model partitioning algorithm to get the optimal minimum execution time using the dynamic programming technique with the profiled data. We evaluated our work by executing the BERT, RoBERTa, and GPT-2 models on the ARM multicores with the PIM-modeled FPGA platform with various sequence lengths. For three computing devices in the platform, i.e., CPU serial/parallel and PIM executions, we could find all the costs only in four profile runs, three for node costs and one for edge costs. Also, our model partitioning algorithm achieved the highest performance in all the experiments over the execution with manually assigned device priority and the state-of-the-art greedy approach.

Accelerating Multiple Sequence Alignments Using Parallel Computing

Multiple sequence alignment (MSA) stands as a critical tool for understanding the evolutionary and functional relationships among biological sequences. Obtaining an exact solution for MSA, termed exact-MSA, is a significant challenge due to the combinatorial nature of the problem. Using the dynamic programming technique to solve MSA is recognized as a highly computationally complex algorithm. To cope with the computational demands of MSA, parallel computing offers the potential for significant speedup in MSA. In this study, we investigated the utilization of parallelization to solve the exact-MSA using three proposed novel approaches. In these approaches, we used multi-threading techniques to improve the performance of the dynamic programming algorithms in solving the exact-MSA. We developed and employed three parallel approaches, named diagonal traversing, blocking, and slicing, to improve MSA performance. The proposed method accelerated the exact-MSA algorithm by around 4×. The suggested approaches could be basic approaches to be combined with many existing techniques. These proposed approaches could serve as foundational elements, offering potential integration with existing techniques for comprehensive MSA enhancement.

ОПТИМІЗАЦІЯ ЛОГІСТИКИ ТА РЕАЛІЗАЦІЇ ПРОДУКЦІЇ В СІЛЬСЬКОГОСПОДАРСЬКИХ ПІДПРИЄМСТВАХ НА УМОВАХ КООПЕРАЦІЇ

This extensive article delves into the critical importance of optimizing transport logistics within the agricultural sector, providing a thorough examination of key algorithms and advanced techniques aimed at significantly improving industry efficiency. The primary focus is dedicated to the meticulous selection of optimal routes, involving a detailed review of state-of-the-art routing algorithms and dynamic programming techniques. The article systematically guides readers through concrete steps for determining optimal delivery routes, considering a myriad of factors such as distance, road conditions, transport volume, type, delivery deadlines, and transportation costs. The complexity of this task necessitates an in-depth analysis of various elements, and the article provides valuable insights into how these considerations collectively contribute to the overall efficiency of agricultural transport logistics. Emphasizing the pivotal role of effective resource management, the article underscores the need for strategic allocation, transportation, and utilization of diverse resources, including transport assets, human resources, equipment, time, and more. The integration of modern technologies is identified as a crucial component for achieving optimization goals, ensuring real-time data analysis, efficient communication, and adaptive decision-making. As a practical outcome, the article offers strategic recommendations for the implementation of optimization approaches within the agricultural logistics framework. These recommendations encompass innovative solutions for route planning, resource allocation, and technology integration. The overarching goal is to enhance the viability and productivity of agriculture, positioning logistical optimization as a key driver for sustainable growth in the industry. The synthesis of algorithmic strategies and cutting-edge technologies is positioned as instrumental in fostering advancements within the agricultural landscape. The article serves as a comprehensive guide, catering to various stakeholders, including farmers, logistics professionals, and policymakers, who aspire to leverage logistical optimization for transformative outcomes in agriculture. In conclusion, this in-depth exploration not only highlights the immediate benefits of optimizing transport logistics but also underscores the long-term sustainability and growth potential it brings to the agricultural sector. The integration of advanced algorithms and technologies is positioned as a cornerstone for addressing the evolving challenges and demands of modern agriculture.

A single player and a mass of agents: A pursuit evasion-like game

We study a finite-horizon differential game of pursuit-evasion like, between a single player and a mass of agents. The player and the mass directly control their own evolution, which for the mass is given by a first order PDE of transport equation type. Using also an adapted concept of non-anticipating strategies, we derive an infinite dimensional Isaacs equation, and by dynamic programming techniques we prove that the value function is the unique viscosity solution on a suitable invariant subset of a Hilbert space.