Dynamic Programming Problem Research Articles

Dynamic Plan Control: An Effective Tool to Manage Demand Considering Mobile Internet Network Congestion

Rapidly increasing mobile data traffic have placed a significant burden on mobile Internet networks. Due to limited network capacity, a mobile network is congested when it handles too much data traffic simultaneously. In turn, some customers leave the network, which induces a revenue loss for the mobile service provider. To manage demand and maximize revenue, we propose a dynamic plan control method for the mobile service providers under connection-speed-restriction pricing. This method allows the mobile service provider to dynamically set the data plans’ availability for potential customers’ new subscriptions. With dynamic plan control, the service provider can adjust data network utilization and achieve high customer satisfaction and a low churn rate, which reflect high service supply chain performance. To find the optimal control policy, we transform the high-dimensional dynamic programming problem into an equivalent mixed integer linear programming problem. We find that dynamic plan control is an effective tool for managing demand and increasing revenue in the long term. Numerical evaluation with a large European mobile service provider further supports our conclusion. Furthermore, when network capacity or potential customers’ willingness to join the network changes, the dynamic plan control method generates robust revenue for the service provider.

A Hilbert transform approach for controlled jump-diffusions with financial applications

We propose a new computational method for a class of controlled jump-diffusions for financial applications. In the first step of our method, we apply piecewise constant policy approximation where we partition the time horizon into small time intervals and the control is constant on each interval. In the second step, we develop a Hilbert transform approach to solve a discrete time dynamic programming problem. We provide rigorous error bounds for the piecewise constant policy approximation for controlled jump-diffusions, generalizing previous results for diffusions. We also apply our method to solve two classical types of financial problems: option pricing under uncertain volatility and/or correlation models and optimal investment, including utility maximization and mean-variance portfolio selection. Through various numerical examples, we demonstrate the properties of our method and show that it is a computationally efficient choice for low-dimensional problems. Our method also compares favorably with some popular approaches.

Online Planning with Offline Simulation

One of the central issues in (finite horizon) online planning problems is to synthesize the impact of real time decisions on the subsequent states of the system, and the performance in the remaining time horizon (cost-to-go function). A complete resolution often leads to intractable dynamic programming problems. In this paper, we propose a computationally efficient approach to this problem that attains near-optimal performance in non-stationary environments. More specifically, we study a general class of online planning problems with concave objective functions and (global) feasibility constraints. A wide range of problems in supply chain management, online advertising, and network revenue management etc., can be appropriately modelled using this online planning framework. Leveraging on the value of the gradient information obtained from offline simulation (generated from the distributional information), we develop a generic approach to facilitate online planning for this class of problems. Furthermore, our proposed approach produces near optimal solution with sublinear regret and satisfies the feasibility constraints with high probability. We present extensive numerical evidence to validate the performance of this approach, and discuss its improvement over existing techniques that assume the underlying environment is stationary.

Several Fixed-Point Theorems forF-Contractions in Complete Branciarib-Metric Spaces and Applications

In this paper, we prove the existence and uniqueness of fixed points forF-contractions in complete Branciarib-metric spaces. Furthermore, an example for supporting the related result is shown. We also present the concept of the weak well-posedness of the fixed-point problem of the mappingTand discuss the weak well-posedness of the fixed-point problem of anF-contraction in complete Branciarib-metric spaces. Besides, we investigate the problem of common fixed points forF-contractions in above spaces. As an application, we apply our main results to solving the existence and uniqueness of solutions for a class of the integral equation and the dynamic programming problem, respectively.

Intertemporal Pricing via Nonparametric Estimation: Integrating Reference Effects and Consumer Heterogeneity

We consider intertemporal pricing in the presence of reference effects and consumer heterogeneity. Our research question encompasses how to estimate heterogeneous consumer reference effects from data and how to efficiently compute the optimal pricing policy. Understanding reference effects is essential for designing pricing policies in modern retailing. Our work contributes to this area by further incorporating consumer heterogeneity under arbitrary distributions. We propose a demand model that allows arbitrary joint distributions of valuations, responsiveness to prices, and responsiveness to reference prices among consumers. To learn consumer heterogeneity from transaction data, we use a nonparametric estimation method. We formulate the pricing optimization as an infinite horizon dynamic programming problem and solve it by applying a modified policy iteration algorithm. We investigate the structure of optimal pricing policies and prove the sub-optimality of constant pricing policies even when all consumers are loss-averse according to the classical definition. Our numerical studies further show that our estimation and optimization framework improves the expected revenue of retailers via accounting for heterogeneity. We validate our model using real data from JD.com, a large E-commerce retailer, and find empirical evidence of consumer heterogeneity. In practice, ignoring consumer heterogeneity may lead to a significant loss of revenue. Furthermore, the existence of heterogeneous reference effects offers a strong motive for promotions and price fluctuations.

Multiagent value iteration algorithms in dynamic programming and reinforcement learning

We consider infinite horizon dynamic programming problems, where the control at each stage consists of several distinct decisions, each one made by one of several agents. In an earlier work we introduced a policy iteration algorithm, where the policy improvement is done one-agent-at-a-time in a given order, with knowledge of the choices of the preceding agents in the order. As a result, the amount of computation for each policy improvement grows linearly with the number of agents, as opposed to exponentially for the standard all-agents-at-once method. For the case of a finite-state discounted problem, we showed convergence to an agent-by-agent optimal policy. In this paper, this result is extended to value iteration and optimistic versions of policy iteration, as well as to more general DP problems where the Bellman operator is a contraction mapping, such as stochastic shortest path problems with all policies being proper.

Discrete-time MPC for switched systems with applications to biomedical problems

This paper studies switched systems in which the manipulated control action is the time-depending switching signal. To control the switched systems means to select an autonomous system - at each time step - among a given finite family. Even when this selection can be done by solving a Dynamic Programming (DP) problem, such a solution is often difficult to apply, and state/control constraints cannot be explicitly considered. In this work a new set-based Model Predictive Control (MPC) strategy is proposed to handle switched systems in a tractable form. The optimization problem at the core of the MPC formulation consists in an easy-to-solve mixed-integer optimization problem, whose solution is applied in a receding horizon way. Applications to schedule therapies in viral infection and cancer treatments are studied. The numerical results suggest that the proposed strategy outperforms the schedule for available treatments.

Editorial Special Issue on Adaptive Dynamic Programming and Reinforcement Learning

The past decade has witnessed a surge in research activities related to adaptive dynamic programming (ADP) and reinforcement learning (RL), particularly for control applications. Several books [item 1)–5) in the Appendix] and survey papers [item 6)–10) in the Appendix] have been published on the subject. Both ADP and RL provide approximate solutions to dynamic programming problems. In a 1995 article by Barto <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">et al.</i> [item 11) in the Appendix], they introduced the so-called “adaptive real-time dynamic programming,” which was specifically to apply ADP for real-time control. Later, in 2002, Murray <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">et al.</i> [item 12) in the Appendix] developed an ADP algorithm for optimal control of continuous-time affine nonlinear systems. On the other hand, the most famous algorithms in RL are the temporal difference algorithm [item 13) in the Appendix] and the Q-learning algorithm [item 14) and 15) in the Appendix].

Approximate dynamic programming‐based decentralised robust optimisation approach for multi‐area economic dispatch considering wind power uncertainty

This study presents a fully decentralised robust optimisation (RO) approach for multi-area economic dispatch (MA-ED) in the presence of wind power uncertainty. Unlike traditional algorithms, the authors formulate this MA-ED problem as dynamic programming problem, and decompose the centralised robust MA-ED problem into a series of sub-problems based on approximate dynamic programming algorithm. The value functions are proposed for each area to iteratively estimate the impacts of its dispatches on the dispatches of other areas which make decisions subsequently. The proposed algorithm does not require a central operator but only needs to exchange a small amount of information among neighbouring areas to achieve fully decentralised decision-making. It is practical in cases where the centralised operator cannot be implemented considering the dispatch independence and the detailed data of one area is unavailable considering the privacy. Additionally, the accuracy, adaptability and computational efficiency of the proposed algorithm are illustrated using numerical simulations on two test systems and an actual power system.

Easily solving dynamic programming problems in Haskell by memoization of hylomorphisms

SummaryDynamic programming is a well‐known algorithmic technique that solves problems by a combination of dividing a problem into subproblems and using memoization to avoid an exponential growth of the costs. We show how to implement dynamic programming in Haskell using a variation of hylomorphisms that includes memoization. Our implementation uses polymorphism so the same function can return the best score or the solution to the problem based on the type of the returned value.

Routing in Communication Networks With Interdependent Power Grid

In smart grid, power grid and communication network are interdependent. With this interdependence, a failure in one network may lead to a cascade of failures across the networks. Considering such failures, we develop a model to determine communication route failure probability and quantify its impact in term of lost load. We formulate an optimization to find the set of optimal routes that minimize the impact triggered by an initial route failure. Unfortunately, the optimization is NP-hard and relies on a non-trivial algorithm to calculate the amount of lost load. We further transform the optimization into a dynamic programming problem using the Bellman’s optimality equation. We propose a Gauss-Seidel value iteration algorithm to solve the dynamic programming problem. Simulation results confirm that the proposed algorithm can accurately find the set of optimal routes. For a network configuration with 7 power nodes and 22 routers, the algorithm can efficiently find optimal routes in 2.4 seconds while solving the original optimization takes more than 48 hours.

Synthesis of Large-Scale Supply Networks for Complete Long-term Transition from Fossil to Renewable-based Production of Energy and Bioproducts

This contribution presents the synthesis of sustainable larger-scale supply networks to produce alternative energy and bioproducts across an entire system’s lifetime. In particular, production of food, first, second and third generation of biofuels, hydrogen, and renewable electricity are considered. Corn and wheat grain are used for production of food; corn stover, wheat straw, miscanthus, forest residue, algae oil and waste cooking oil are biomass and waste sources for the manufacture of bioproducts; and solar, wind and geothermal sources for electricity production. The objective of supply network synthesis is the maximization of Sustainability Net Present Value, which is a sustainability metric composed of economic, environmental and social net present values. The dynamic mixed-integer linear programming problem is formulated as a multi-period model where i) monthly time periods are used for the manufacture of biomass and bioproducts, while ii) daily, hourly and monthly periods are used for the production of electricity. A case study of EU-27 is considered and gradual energy transition from the current energy supply to 100 % renewable-based production of fuels and electricity over a time horizon of 40 y is investigated. The results of this study indicate that long-term transition from fossil to renewable-based generation of electricity, fuels and bioproducts is possible to a large extent. The results suggest an optimal selection of types, locations and dynamics of technology installations across the EU in order to achieve energy transition in the transport and electricity sectors.

Speed-up credit exposure calculations for pricing and risk management

We introduce a new method to calculate the credit exposure of European and path-dependent options. The proposed method is able to calculate accurate expected exposure and potential future exposure profiles under the risk-neutral and the real-world measure. A key advantage is that it delivers an accuracy comparable to a full re-evaluation and at the same time it is faster than a regression-based method. The core of the approach is solving a dynamic programming problem by function approximation. This yields a closed-form approximation along the paths together with the option's delta and gamma. The simple structure allows for highly efficient evaluation of the exposures, even for a large number of simulated paths. The approach is flexible in the model choice, payoff profiles and asset classes. We validate the accuracy of the method numerically for three different equity products and a Bermudan interest rate swaption. Benchmarking against the popular least-squares Monte Carlo approach shows that our method is able to deliver a higher accuracy in a faster runtime.

Existence, uniqueness and well-posedness results for relation theoretic coupled fixed points problem using C- class function with some consequences and an application

In this paper we consider a coupled fixed point problem where the mapping is supposed to satisfy an inequality involving a special type of function which is familiarly known as $$\mathbf{C }$$ class function and contains rational terms. We first prove the existence of a coupled fixed point of such mappings. Then we show that the coupled fixed point is unique under some additional conditions. We investigate the well-posedness of the problem. Our results are derived in metric spaces with an appropriate relation. The inequality is supposed to be valid only for pairs of points appropriately related by the binary relation. We discuss some consequences and an application of our result to a problem of a functional equation relevant to dynamic programming problems. The main result is illustrated through an example.

Perturbed Manipulability Optimization in a Distributed Network of Redundant Robots

For avoiding a singularity arising in the cooperative control of multiple redundant robot manipulators, an efficient way is to maximize the manipulability. In this article, by making progress along this direction, a distributed manipulability optimization scheme is proposed to maximize the manipulability of redundant robot manipulators in a distributed network with limited communication. With manipulability optimization incorporated in the proposed scheme, all the involved manipulators can be regulated to track their optimal configurations dynamically, in addition to the collaboration among them to complete the specified tasks. To do this, the distributed scheme is transformed into a dynamic quadratic programming (QP) problem by considering the time dependence of the parameters. Then, a generalized recurrent neural network (GRNN) is constructed and proposed to deal with the QP problem online with perturbations considered. Theoretical analysis is conducted, which confirms that the proposed GRNN is able to globally converge to the optimal solution to the dynamic QP problem in the presence of noises and perturbations. Finally, simulation results based on a distributed network of redundant robots are conducted and presented to verify the superior performance of the proposed distributed manipulability optimization scheme.

Tree Edit Distance Cannot be Computed in Strongly Subcubic Time (Unless APSP Can)

The edit distance between two rooted ordered trees with n nodes labeled from an alphabet Ʃ is the minimum cost of transforming one tree into the other by a sequence of elementary operations consisting of deleting and relabeling existing nodes, as well as inserting new nodes. Tree edit distance is a well-known generalization of string edit distance. The fastest known algorithm for tree edit distance runs in cubic O ( n 3 ) time and is based on a similar dynamic programming solution as string edit distance. In this article, we show that a truly subcubic O ( n 3-ε ) time algorithm for tree edit distance is unlikely: For |Ʃ| = Ω ( n ), a truly subcubic algorithm for tree edit distance implies a truly subcubic algorithm for the all pairs shortest paths problem. For |Ʃ| = O (1), a truly subcubic algorithm for tree edit distance implies an O ( n k-ε ) algorithm for finding a maximum weight k -clique. Thus, while in terms of upper bounds string edit distance and tree edit distance are highly related, in terms of lower bounds string edit distance exhibits the hardness of the strong exponential time hypothesis (Backurs, Indyk STOC’15) whereas tree edit distance exhibits the hardness of all pairs shortest paths. Our result provides a matching conditional lower bound for one of the last remaining classic dynamic programming problems.

Energy-Efficient Subway Train Scheduling Design With Time-Dependent Demand Based on an Approximate Dynamic Programming Approach

Owing to environmental concerns, the energy-efficient subway train scheduling problem is necessary in subway operation management. This paper designs an approximate dynamic programming (DP) approach for energy-efficient subway train scheduling problem with time-dependent demand. The train traffic model is proposed with the dynamic equations for the evolution of train headway, train passenger loads, and the energy consumption along the subway line. For the dynamic changing of the onboard passengers with time, the total train energy usage is modeled as the sum of energy consumptions from the traction system and auxiliary facilities. A nonlinear DP problem is formulated to generate a near optimal timetable to realize the tradeoff among the utilization of trains, passenger waiting time, service levels, and energy consumption. To overcome the curse of dimensionality in this optimization problem, we construct an approximate DP framework, where the conceptions of states, policies, state transitions, and reward function are introduced. And this algorithm is able to converge to a good solution with a short time compared to the genetic algorithm and differential evolution algorithm. Finally, the numerical experiments are given to demonstrate the effectiveness of the proposed model and algorithm.

Extensions of the Dynamic Programming Framework: Battery Scheduling, Demand Charges, and Renewable Integration

We consider a general class of dynamic programming (DP) problems with nonseparable objective functions. We show that for any problem in this class, there exists an augmented-state DP problem that satisfies the principle of optimality and the solutions to which yield solutions to the original problem. Furthermore, we identify a subclass of DP problems with naturally forward separable objective functions for which this state-augmentation scheme is tractable. We extend this framework to stochastic DP problems, proposing a suitable definition of the principle of optimality. We then apply the resulting algorithms to the problem of optimal battery scheduling with demand charges using a data-based stochastic model for electricity usage and solar generation by the consumer.

The Lipschitz Properties of the Value Function and the Solution Map to a Parametric Discrete Optimal Control Problem

Motivated by the works of Toan and Kien (J. Nonlinear Convex Anal.12:635–650, 2011) and Thuy and Toan (Acta Math. Vietnam.43:175–199, 2018) on the stability and the solution sensitivity in parametric discrete optimal control problems, this paper is devoted to the study of the Lipschitz property of the value function and the Aubin property of the solution map to a parametric dynamic programming problem with linear constraints. By establishing abstract results on the Lipschitz property of the value function to a parametric programming and the Aubin property of the solution map to a parametric variational inequality, we obtain the Lipschitz property of the value function and the Aubin property of the solution map to a parametric discrete optimal control problem.

Search for all solutions to the dynamic programming problem if their multicriteria estimates coincide

Among the mathematical methods used in economics, a prominent place is occupied by the dynamic programming method, with the help of which the optimal control of multi-stage processes is organized. The disadvantage of this method is the impossibility of calculating all solutions to the problem if their criteria-based estimates coincide. The fact of the existence of several optimal trajectories of a multi-step process may mean that the task is not set correctly, in the sense that the assigned criteria do not fully characterize the system under study. This means that the traditional method of dynamic programming needs to be refined in case of the existence of several optimal trajectories with the same value of the criterion. This article proposes the most general version of such refinement, namely, a multi-criteria numerical scheme is generalized. For a more visual representation of calculations and the result of the study, we will describe the discrete dynamic programming problem in terms of graph theory. In this case, it reduces to the problem of finding the optimal path on a directed graph. To solve it, a three-stage algorithm is proposed, the composition of which includes the following steps. The first stage is the construction of optimal criteria estimates for paths from the initial vertex to all the others. To perform this stage, the most universal method is the multicriteria version of the Ford – Bellman method. The second stage is the construction of a graph of optimal paths. In the original graph, arcs are selected that are part of the optimal paths. Of these, using the original algorithm, a subgraph is formed in which all paths are optimal. It is analytically proved that this algorithm gives the correct result (correct). The third stage is enumeration of all paths in the constructed subgraph. Numerical experiments showed that the proposed three-stage method works efficiently on oriented graphs of any type in a sufficiently large range of dimensions. The proposed algorithm with minimal changes can be used to solve an arbitrary discrete dynamic programming problem.