Dynamic Programming Problem Research Articles

Information Relaxation and a Duality-Driven Algorithm for Stochastic Dynamic Programs

The curse of dimensionality significantly restricts the use of dynamic programming methods in solving complex problems. Consequently, researchers and practitioners often resort to approximate (suboptimal) control policies that strike a balance between ease of implementation and satisfactory performance. Information relaxation-based duality techniques generate both upper and lower bounds for the true values of stochastic dynamic programming problems, allowing us to evaluate the optimality of approximate policies through the dual gap. However, the literature still lacks guidance on handling cases where the gaps are excessively loose. In “Information Relaxation and a Duality-Driven Algorithm for Stochastic Dynamic Programs,” Chen, Ma, Liu, and Yu develop a novel DDP framework to obtain and tighten confidence interval estimates for the true value functions of SDP problems. Leveraging a new finding that the dual operation yields subsolutions, they establish convergence guarantees for DDP. Additionally, a regression-based Monte Carlo method is introduced, aimed at high-dimensional applications. Numerical examples demonstrate that DDP effectively improves dual gaps for various heuristics that are commonly used in the literature.

Optimal trading with regime switching: Numerical and analytic techniques applied to valuing storage in an electricity balancing market

Accurately valuing storage in the electricity market recognizes its role in enhancing grid flexibility, integrating renewable energy, managing peak loads, providing ancillary services and improving market efficiency. In this paper we outline an optimal trading problem for an Energy Storage Device trading on the electricity balancing (or regulating) market. To capture the features of the balancing (or regulating) market price we combine stochastic differential equations with Markov regime switching to create a novel model, and outline how this can be calibrated to real market data available from NordPool. By modelling a battery that can be filled or emptied instantaneously, this simplifying assumption allows us to generate numerical and quasi analytic solutions.We implement a case study to investigate the behaviour of the optimal strategy, how it is affected by price and underlying model parameters. Using numerical (finite-difference) techniques to solve the dynamic programming problem, we can estimate the value of operating an Energy Storage Device in the market given fixed costs to charge or discharge. Finally we use properties of the numerical solution to propose a simple quasi-analytic approximation to the problem. We find that analytic techniques can be used to give a benchmark value for the storage price when price variations during the day are relatively small.

A zeroing feedback gradient-based neural dynamics model for solving dynamic quadratic programming problems with linear equation constraints in finite time

A zeroing feedback gradient-based neural dynamics model for solving dynamic quadratic programming problems with linear equation constraints in finite time

Enhancing Make-to-Order Manufacturing Agility: When Flexible Capacity Meets Dynamic Pricing

The rise of online marketplaces has raised customer expectations regarding customization and lead time. It poses significant challenges to manufacturing firms and prompts a move from make-to-stock to a more flexible make-to-order system. Compared to make-to-stock settings, make-to-order systems cannot smooth fluctuations in demand using available stock. While viewing dynamic pricing as a useful strategy to balance supply with demand, many manufacturing firms can also create capacity flexibility. In that scenario, system costs could be cut by managing capacity and demand simultaneously. In this paper, we consider a make-to-order production environment with base and surge capacity as well as the ability to adjust product pricing. Our main focus is on operational decision-making, assuming that the base capacity and surge capacity are fixed, but activating the surge capacity incurs a setup cost. Initially, we propose a stochastic control model to reflect this complex decision problem. However, our initial model leads to an intractable dynamic programming problem. To overcome this, we convert the problem to a more tractable diffusion control problem. This approach helps to reveal the conditions under which utilizing flexible capacity is more advantageous than relying solely on fixed capacity. When flexible capacity is advantageous, we provide a solution to the diffusion control problem that can guide optimal capacity and price adjustments. We discover an interesting interplay between capacity adjustment and dynamic pricing. In particular, we find that the price, which aims at reducing congestion, may not monotonically increase with the congestion level when capacity adjustments incur a fixed cost.

Artificial neural networks to solve dynamic programming problems: A bias-corrected Monte Carlo operator

Artificial Neural Networks (ANNs) are powerful tools that can solve dynamic programming problems arising in economics. In this context, estimating ANN parameters involves minimizing a loss function based on the model's stochastic functional equations. In general, the expectations appearing in the loss function admit no closed-form solution, so numerical approximation techniques must be used. In this paper, I analyze a bias-corrected Monte Carlo operator (bc-MC) that approximates expectations by Monte Carlo. I show that the bc-MC operator is a generalization of the all-in-one expectation operator, already proposed in the literature. I demonstrate that, under some conditions on the primitives of the economic model, the bc-MC operator is the unbiased estimator of the loss function with the minimum variance. I propose a method to optimally set the hyperparameters defining the bc-MC operator, and illustrate the findings numerically with well-known economic models. I also demonstrate that the bc-MC operator can scale to high-dimensional models. With just approximately a minute of computing time, I find a global solution to an economic model with a kink in the decision function and more than 100 dimensions.

Improvement of the method of time rationing for assembling car groups on one track

Purpose. To improve the method for standardizing the duration of the shunting operation of assembling cars on one track. This can be achieved as a result of solving the following research problems: development of a method for searching the optimal order of assembling cars on one track; distribution parameters estimation of the random value of the duration of shunting operation of assembling cars on one track based on calculation experiments. Methodology. During the research, the methods of theory of railway operation, dynamic programming and mathematical statistics were used. Findings. Research on the assembling process of car group to one track established the distribution parameters of the random variable of time spent for shunting. In the course of the research, the problem of choosing the optimal order of shunting operations during car assembling was formalized and solved as a problem of dynamic programming. The time spent for shunting work was chosen as the optimality criterion. The paper considers the possibility of approximating the data of calculation experiments by analytical dependencies. It was found out that the use of linear polynomials with interaction allows obtaining dependencies describing time standards with a relative accuracy of ±5 %. Originality. The method is improved for developing the time standards for shunting work, which, unlike the existing one, is based on the performance of a series of calculation experiments, each of which solves the optimization problem of finding such an order of assembling cars that ensures minimum time consumption for shunting. Practical value. The methods developed in the work and the dependencies obtained allow improving the quality of decisions made when developing technology and designing railway stations and sidings of industrial enterprises.

A kernel-based approximate dynamic programming approach: Theory and application

This article proposes a novel kernel-based Dynamic Programming (DP) approximation method to tackle the typical curse of dimensionality of stochastic DP problems over the finite time horizon. Such a method utilizes kernel functions in combination with Support Vector Machine (SVM) regression to determine an approximate cost function for the entire state space of the underlying Markov Decision Process (MDP), by leveraging cost function computed for selected representative states. Kernel functions are used to define the so-called kernel matrix, while the parameter vector of the given kernel-based cost function approximation is computed by moving backwards in time from the terminal condition and by applying SVM regression. This way, the difficulty of selecting a proper set of features is also tackled. The proposed method is then extended to the infinite time horizon case. To show the effectiveness of the proposed approach, the resulting Recursive Residual Approximate Dynamic Programming (RR-ADP) algorithm is applied to the sensor scheduling design in multi-process remote state estimation problems.

Optimal control for quadrotors during inspection of power utility assets

Renewable energy and power utilities inspection by autonomous aircraft enables rapid and effective risk-free assessment of the state of systems, and provides a qualitative and accurate assessment of defects and damages. To realize maximum operational benefits of aerial inspections, effective controls for autonomous aircraft must be ensured and the system should be operated in an optimal policy. Hence, the objective of this paper is to consider optimal control strategies for a quadrotor helicopter type UAV. Based on some structural properties of the considered system, in particular the flatness property, we suggest a control strategy that ensures tracking a time parameterized path that connects two given points in the state space while minimizing the energy consumption. The proposed controller enables longer-endurance missions for the quadrotor and effective supervision and inspection of large energy systems and power plants. In this paper the analysis of the control law takes into consideration the highly nonlinear dynamic model of the quadrotor and the electrical actuator model. The current approach in solving the optimal control problem under consideration allows one to eliminate the differential equation and to reformulate the optimal control problem to a nonlinear dynamic programming problem. The approach can make monitoring operations of renewable energy plants by aerial vehicles, more efficient.

Methodology for forming a rational structure of technology for the restoration of car, vehicle and road-building machinery parts

The scientific work examines the problem of effective and high-quality restoration of cars, vehicles and road construction machines. The assessment of approaches in determining the method and technology for restoring parts of roadbuilding machines is considered. The decisions at the stage of choosing a method for restoring parts and at the stage of forming the general structure of the technological process are analyzed. The new approach makes it possible to achieve maximum efficiency in structuring the technological process of restoring parts of road-building machines; it is carried out according to a number of well-founded criteria when integrating the zoning method on the principle of observing the ratio of probabilities of possible states of the research environment in solving the problem of dynamic programming. Keywords truck, vehicle, road construction vehicle, part restoration, mathematical model, quality, restoration quality indicators

Neutrosophic parametric study of piecewise quadratic fuzzy multi-objective dynamic programming problems

The goal of this research is to investigate fuzzy multiobjective dynamic programming issues with fuzzy parameters in the objective functions and single valued trapezoidal neutrosophic numbers in the left hand side of the constraints. Piecewise quadratic fuzzy numbers characterize these fuzzy parameters. In addition, applying the score function of the neutrosophic numbers to convert the constraints parameters into its crisp . Some basic notions in the problem under the pareto optimal solution concept is redefined and analyzed to study the stability of the problem. Furthermore, a technique is presented for obtaining a subset of the parametric space that has the same pareto optimal solution. For a better understanding and comprehension of the suggested concept, a numerical example is provided.

Mathematical simulation of combat actions on two encounters using dynamic programming and Wolfram Mathematica package

The work is devoted to the important task of modeling combat operations at various areas of the conflict with the possibility of redistribution of combat resources during the battle. The problem of dynamic programming is formulated with the objective function as a function of the enemy's losses. This function is determined using the system of differential equations of Lanchester in the conditions of "highly organized" combat. In the work, using the Wolfram Mathematica package, a computer implementation of the solution to the problem of finding the optimal number of combat units, which should be distributed by one of the parties at the initial moment of time to the first site of the collision and then transferred from of the first site to the second at some subsequent moment of time with the aim of inflicting maximum losses on the other side (the enemy) at a certain moment of time. Examples of this implementation are given. Conclusions are made based on the sample solutions tables. Firstly, the side having a disadvantage in combat units at the beginning of the battle can destroy a significant part of the enemy due to the optimal distribution of combat units. Secondly, due to the optimal distribution of combat units, it is possible to end the battle by completely destroying the enemy at a certain point of time. The examples have shown the ability of the computers to predict the outcome of the battle at two points of contact. The computer program can be organized to destroy as much of the enemy’s armory as possible before a certain moment of time, or to destroy it completely at a certain moment of time. The results of this work will be used to solve the more general problem of redistributing combat resources across different areas of the conflict (more than two) at different points in time (more than two). In addition, a similar approach can be used when the battle between the parties is "poorly organized" or is a battle with reinforcements, etc.

Approximate dynamic programming for constrained linear systems: A piecewise quadratic approximation approach

Approximate dynamic programming (ADP) faces challenges in dealing with constraints in control problems. Model predictive control (MPC) is, in comparison, well-known for its accommodation of constraints and stability guarantees, although its computation is sometimes prohibitive. This paper introduces an approach combining the two methodologies to overcome their individual limitations. The predictive control law for constrained linear quadratic regulation (CLQR) problems has been proven to be piecewise affine (PWA) while the value function is piecewise quadratic. We exploit these formal results from MPC to design an ADP method for CLQR problems with a known model. A novel convex and piecewise quadratic neural network with a local–global architecture is proposed to provide an accurate approximation of the value function, which is used as the cost-to-go function in the online dynamic programming problem. An efficient decomposition algorithm is developed to generate the control policy and speed up the online computation. Rigorous stability analysis of the closed-loop system is conducted for the proposed control scheme under the condition that a good approximation of the value function is achieved. Comparative simulations are carried out to demonstrate the potential of the proposed method in terms of online computation and optimality.

ADP-Based Optimal Control of Linear Singularly Perturbed Systems With Uncertain Dynamics: A Two-Stage Value Iteration Method

We study the problem of adaptive dynamic programming (ADP) based on optimal control of linear singularly perturbed systems (SPSs) subject to completely unknown dynamics. Previous works on ADP-based optimal control of SPSs require that the fast dynamics of the system are either known a priori or unknown but asymptotically stable, and such results are limited to standard cases, i.e., the internal dynamics of their fast subsystems are nonsingular. In this brief, these conditions can be removed through a sequential procedure to design feedback gains of both fast and slow states in the framework of ADP. In this procedure, two optimal control problems are formulated for the prefeedback fast subsystem and the modified slow subsystem. Then, a data-driven two-stage value iteration algorithm is imposed to learn the optimal controller without using any system dynamics. Fundamentally different from existing works, an initial admissible control policy is no longer needed during learning. Finally, the effectiveness of the learning algorithm is proved by a simulation example.

Dynamic User-Scheduling and Power Allocation for SWIPT Aided Federated Learning: A Deep Learning Approach

Federated learning (FL) has been considered as a promising paradigm for enabling distributed machine learning (ML) in wireless networks. To address the limited energy capacity of wireless devices, we propose a simultaneous wireless information and power transfer (SWIPT) aided FL, in which one FL server (FLS) co-located at a cellular base station (BS) uses SWIPT to simultaneously broadcast the global model to wireless user-devices (UDs) and provide wireless power transfer to them. The UDs then use the harvested energy to train their local models and further transmit the local models to the FLS for aggregation. To improve the spectrum efficiency, we consider that the UDs form a non-orthogonal multiple access (NOMA) group for simultaneously sending their local models over the same spectrum channel. Taking the UDs' time-varying available energy and channel conditions into account, we propose a dynamic optimization of the UDs-scheduling, the BS's transmit-power allocation, and the UDs' power-splitting factors for SWIPT, with the objective of minimizing the long-term energy consumption while ensuring the FL convergence. The optimization problem, however, is challenging to solve since it is a finite-horizon dynamic programming problem but with an unknown stopping time, and moreover, the action space covers both discrete and continuous variables. To address these difficulties, we first execute a series of equivalent transformations to reduce the number of decision variables and then formulate the problem as a stochastic shortest path problem, based on which we propose an actor-critic deep reinforcement learning algorithm with the proximal policy optimization to efficiently learn the policy that dynamically adjusts the UDs-scheduling for FL as well as the BS's transmit-power for SWIPT. Numerical results validate the effectiveness and performance of our proposed algorithm. The results demonstrate that our proposed algorithm can effectively reduce the long-term energy consumption in comparison with two baseline algorithms.

Optimizing Operations Management and Business Analytics Strategies under Uncertainty: Dynamic Programming

Dynamic programming is a method used in mathematics, management science, computer science, economics, and bioinformatics to solve complex problems by decomposing the original problem into relatively simple sub-problems. Dynamic programming is often applicable to problems with overlapping subproblems and optimal substructure properties. This paper employs a comprehensive research approach of literature review, as well as empirical analysis and case studies to investigate the topic and demonstrate the practicality and effectiveness of dynamic programming in solving complex decision-making problems. However, the curse of dimensionality poses challenges when dealing with high-dimensional decision spaces, requiring approximate dynamic programming methods, including reinforcement learning algorithms. Therefore, when dealing with more complex dynamic programming problems, it is also essential to use program construction tools such as Python to help design a program that can optimize the problem. Therefore, in the learning of dynamic programming, in addition to the correct understanding of basic concepts and methods, specific problems must be analyzed and dealt with in detail, models should be built with rich imagination, and solutions should be solved with creative skills.

Health-aware coordinate long-term and short-term operation for BESS in energy and frequency regulation markets

The penetration of renewable energy in modern power systems is still increasing. Battery energy storage can rapidly respond to a dispatch order and is expected to provide multiple auxiliary services. However, deep charging cycles have negative impacts on battery health. This paper presents a health-aware long-term operation strategy for lithium-ion battery energy storage participating in the energy and frequency regulation markets. The strategy determines the capacity bounds that can be bid in the energy and frequency regulation markets and updates these bounds every three months, aiming to preserve battery health and increase market revenue. A long-term operational modeling framework is proposed to address the multi-timescale nature, integrating frequency control, energy arbitrage, and the evolution of battery degradation in a holistic model. A nonlinear degradation model is developed to approximate the health impact of main stress factors, which captures the nonlinearity of three-stage capacity degradation process caused by the formation of solid electrolyte interphase film and lithium plating. For intraday operation, a two-scale stochastic programming model is proposed, in which the market bidding and automatic generation control response are simulated in the timescales of one hour and two seconds, respectively. This simulation based method is closer to industrial practice, as it accounts for various factors in BESS operation. For the seasonal update of the capacity allocation strategy, a dynamic programming problem is established and solved based on the nonlinear degradation model and the daily revenue obtained from simulation of intraday operation; the capacity allocation strategy is determined from the renowned Principle of Optimality by Bellman. This two timescale modeling framework captures the interaction between short-term operation strategy and long-term degradation process. Numerical simulations validate that the proposed method can slow down battery degradation and increase lifetime revenue.

An improved slope-based adaptive control vector parameterization method for dynamic programming

Control vector parameterization method is the most commonly used numerical computation method in solving dynamic programming problems. However, to balance the expected trajectory and the computational cost, this method faces difficulties in dividing the optimal discretization time grid. In this paper, we propose an effective control approach for non-uniform adaptive grid division. By analyzing the slope change trend of the control parameter, time nodes are adaptively refined by merging time grids with gentle slopes to remove unnecessary time nodes and adding time nodes to time grids with steep slopes to improve function approximation accuracy. Eventually, we obtain an adaptive time grid division method under which the trajectory of discretization control parameter is more approximate to the optimal control trajectory. Under the condition of intuitive slope information, the proposed method can achieve better performance with fewer optimization parameters and shorter computation time. Finally, the proposed method is applied to solve a classic optimal control problem and the obtained results are compared with the traditional control vector parameterization method. By comparison through the example, our proposed method can overcome the contradiction between approximation accuracy and computational cost in control vector parameterization method and improve the optimization efficiency.

Coarse-grained multicomputer parallel algorithm using the four-splitting technique for the minimum cost parenthesizing problem

Dynamic programming is a technique widely used to solve several combinatory optimization problems. A well-known example is the minimum cost parenthesizing problem (MPP), which is usually used to represent a class of non-serial polyadic dynamic-programming problems. These problems are characterized by a strong dependency between subproblems. This paper outlines a coarse-grained multicomputer parallel solution using the four-splitting technique to solve the MPP. It is a partitioning technique consisting of subdividing the dependency graph into subgraphs (or blocks) of variable size and splitting large-size blocks into four subblocks to avoid communication overhead caused by a similar partitioning technique in the literature. Our solution consists in evaluating a block by computing and communicating each subblock of this block to reduce the latency time of processors which accounts for most of the global communication time. It requires O(n^3/p) execution time with O(k * \sqrt{p}) communication rounds. n is the input data size, p is the number of processors, and k is the number of times the size of blocks is subdivided.

Approximate dynamic programming approach to efficient metro train timetabling and passenger flow control strategy with stop-skipping

Urban metro systems continuously face high travel demand during rush hours, which brings excessive energy waste and high risk to passengers. In order to alleviate passenger congestion, improve train service levels and reduce energy consumption, a nonlinear dynamic programming (DP) model of efficient metro train timetabling and passenger flow control strategy with stop-skipping is presented, which consists of state transition equations concerning train traffic and passenger load. To overcome the curse of dimensionality, the formulated nonlinear DP problem is transformed into a discrete Markov decision process, and a novel approximate dynamic programming (ADP) approach is designed based on the lookahead policy and linear parametric value function approximation. Finally, the effectiveness of this method is verified by three groups of numerical experiments. Compared with Particle Swarm Optimization (PSO) and Simulated Annealing (SA), the designed ADP approach could obtain high-quality solutions quickly, which makes it applicable to the practical implementation of metro operations.

Radio Map-Based Trajectory Design for UAV-Assisted Wireless Energy Transmission Communication Network by Deep Reinforcement Learning

In this paper, we consider a wireless energy-carrying communication network of a UAV. In this communication network, the internet of things (IoT) devices maintain their work via the power supply of batteries. The energy of batteries is slowly consumed over time. The UAV adopts the full-duplex working mode and the flight hover protocol, that is, they can power the target device in the hover position and collect base station data at the same time. Unlike traditional methods, which seek to achieve wireless energy transmission, this paper adopts deep reinforcement learning. On the one hand, the deep reinforcement learning algorithm seeks to solve the dynamic programming problem without a model. The traditional method often requires a prior channel model, to form a formula about variables for convex optimization, while reinforcement learning only requires the interaction between agents and the environment. Then, the strategy is optimized according to the reward function feedback. On the other hand, traditional optimization methods generally solve static programming problems. Since IoT devices constantly collect information from the surrounding physical environment, and their requirements for power supply change dynamically, traditional methods are relatively complex and require huge computational overhead, while deep reinforcement learning performs well in complex problems. The purpose of our work is that with the assistance of the radio map, a UAV can find the best hover position, maximize the energy supplied by the UAV, maximize the data throughput collected, and minimize the energy consumption. The simulation results show that the proposed algorithm can well find the best hovering position of the UAVs and significantly improve the performance of the UAV-assisted wireless energy transmission communication network.