Dynamic Programming Problem Research Articles

Z-relation-based multistage decision making

Multistage decision problems have attracted interest in various fields. Starting from the pioneering work of Bellman and Zadeh, a large class of comprehensive studies for fuzzy, stochastic and other settings were proposed. The main issue missed in the existing works is partial reliability of information. Indeed, partial reliability of information about states and actions is important for multistage decisions. Particularly, information about future state of a system resulting from current strategies is partially reliable due to uncertainty. To account for partial reliability aspect, Zadeh introduced the concept of a Z-number. In this paper, we use this concept to consider multistage decision making under partially reliable information. States, strategies, and goals are described by using Z-numbers. State transition matrix that describes evolution of a system in time is formed by using Z-number-based relations. This description leads to a problem of dynamic programming in Z-environment. We use Z-numbers to account for the fact that information related to state transitions is partially reliable – we are partially sure in a result of each transition. The reason behind partial reliability is complexity and uncertainty of real-world problems. A typical example and an application study are used to illustrate the approach.

Reinforcement Learning Powered Station Parameter Optimized Radiation Therapy (SPORT): A Novel Treatment Planning and Beam Delivery Technique

Conventional intensity modulated radiation therapy (IMRT) with a typical 5-20 fixed beams often does not provide sufficient angular sampling required for conformal dose shaping, whereas current volumetric modulated arc therapy (VMAT) discretizes the angular space into equally spaced control points without considering the differential need for intensity modulation of different angles, leading to undersampling at some angles while oversampling at some other angles. Our goal is to develop a node or station parameter optimized radiation therapy (SPORT) strategy with simultaneously optimized angular sampling and beam modulation by leveraging state-of-the-art reinforcement learning and the unique capability of modern digital LINACs in dose delivery through programmable nodal points. We developed a SPORT optimization framework, in which, the process of programming control points (or station parameters) was formulated as a stochastic dynamic programming problem, which was solved by a reinforcement learning-based algorithm. On-policy reinforcement learning method, namely, state-action-reward-state-action (SARSA) was integrated with deep convolutional neural network to predict station parameters by utilizing the patient's anatomical structures meanwhile considering the delivery capability of a typical digital LINAC machine. Here, the deep convolutional neural network estimated the state-action value by using the quality of the plan with current station parameters when a next potential station parameter was selected. The state-action value was then updated by SARSA learning. The quality of the plan was quantified by dosimetry constraints. The model was assessed by a retrospective study on a cohort of patients underwent head-and-neck radiation therapy. Dosimetric analysis and delivery efficiency comparisons were used to evaluate the performance of the proposed framework. Our model was used to generate 16 plans unseen in the original training set. All the plans predicted by our model achieved better dose distributions without violating clinical planning constraints. Moreover, instead of using 4 full standard arcs in the original clinically used plans obtained via manual optimization, the predicted plans only used one full standard arc (about 178 control points) plus boost from a few sub-arcs (less than 30 degrees of gantry angles), which significantly improved the efficiency of the beam delivery. We are in the process of integrating the sub-arcs into the full arc by considering the programmable capability of modern LINACs. We demonstrated that a machine learning-based SPORT framework capable of optimizing the spatial sampling and beam modulation simultaneously for modern radiation therapy. The framework not only significantly improves the quality and efficiency of beam delivery, but also has the potential to be incorporated into current clinical workflow to improve the efficiency of dose planning and delivery.

Long-term optimal coordination of hydro-wind-thermal energy generation using stochastic dynamic programming

ABSTRACT Clean alternative energy and a greater focus on climate change aim to increase the integration of Renewable Energy Sources (RES) into power system networks. As a relatively inexpensive renewable energy, wind energy is integrated into the electrical network to reduce its operating costs. A long-term optimal scheduling model for hydro-wind-thermal in a hybrid generation system is established to find the minimum cost trajectory of energy generation at each period under various constraints. Based on the proposed model and different types of power plants, the original complex problem decomposed into hydro-wind-thermal subproblems. The stochastic Dynamic programming technique (SDP) is employed to solve the complete optimization. In this research, the SDP technique is preferred. This technique handles multistage decision processes by splitting problems down into sequential stages. Because it can incorporate nonlinear and stochastic features into a dynamic programming problem, it has been successful in this hybrid system. A penalty factor was added to the model to reduce outflow variations. As can be seen from the results, outflows are very high during peak demand periods and very low during high inflows. Furthermore, the cost decreases as demand increases, from 40,082.26 $/ GWh in May when demand is 10,275 Gwh to 16,536.32 $/ GWh in January when demand is 17,503 Gwh .

Software for "Asymptotically Optimal Sampling Policy for Selecting Top-m Alternatives"

We consider selecting the top-m alternatives from a finite number of alternatives via Monte Carlo simulation. Under a Bayesian framework, we formulate the sampling decision as a stochastic dynamic programming problem and develop a sequential sam-pling policy that maximizes a value function approximation one-step look ahead. To show the asymptotic optimality of the proposed procedure, the asymptotically optimal sampling ratios that optimize the large deviations rate of the probability of false selection for select-ing the top-m alternatives have been rigorously defined. The proposed sampling policy is not only proved to be consistent but also achieve the asymptotically optimal sampling ratios. Numerical experiments demonstrate superiority of the proposed allocation proce-dure over existing ones.

Asymptotically Optimal Sampling Policy for Selecting Top-m Alternatives

We consider selecting the top-m alternatives from a finite number of alternatives via Monte Carlo simulation. Under a Bayesian framework, we formulate the sampling decision as a stochastic dynamic programming problem and develop a sequential sampling policy that maximizes a value function approximation one-step look ahead. To show the asymptotic optimality of the proposed procedure, the asymptotically optimal sampling ratios that optimize the large deviations rate of the probability of false selection for selecting the top-m alternatives have been rigorously defined. The proposed sampling policy is not only proved to be consistent but also achieve the asymptotically optimal sampling ratios. Numerical experiments demonstrate superiority of the proposed allocation procedure over existing ones. History: Accepted by Bruno Tuffin, Area Editor for Simulation. Funding: This work was supported by the National Natural Science Foundation of China [Grants 72250065, 72293582, 72022001, and 71901003], and the National Science Foundation [Grant DMS-2053489], the major project of the National Natural Science Foundation of China [Grant 72293582], and the China Scholarship Council [Grant CSC202206010152]. Supplemental Material: The software that supports the findings of this study is available within the paper and its Supplemental Information ( https://pubsonline.informs.org/doi/suppl/10.1287/ijoc.2021.0333 ) as well as from the IJOC GitHub software repository ( https://github.com/INFORMSJoC/2021.0333 ). The complete IJOC Software and Data Repository is available at https://informsjoc.github.io/ .

A Heuristic Approach to Explore: The Value of Perfect Information

This research introduces a new heuristic decision model called myopic-value of perfect information (VPI) to study multiarmed bandit (MAB) problems. The myopic-VPI approach only involves ranking the alternatives and computing a one-dimensional integration to obtain the expected future value of exploration. Because myopic-VPI is intuitive and does not involve solving a dynamic programming problem, it has the potential to serve as a useful heuristic approach to model exploration-exploitation tradeoffs. We conduct a series of simulation experiments to study its performance relative to other heuristics under a wide range of parameterizations. We find that myopic-VPI provides significant savings in computational time and decent performance in accumulated utility (although not the strongest) relative to other forward-looking heuristics; this suggests that it is a useful “fast-and-frugal” heuristic. Furthermore, our simulation experiments also reveal the conditions under which myopic-VPI outperforms and underperforms compared with other heuristics. Its empirical performance in the diaper category further shows that myopic-VPI can save estimation time significantly and fit the data on par with index and near-optimal, providing encouraging news that myopic-VPI could be added to the researcher’s or practitioner’s toolkit for MAB problems. This paper was accepted by Gui Liberali, marketing. Supplemental Material: The online appendices are available at https://doi.org/10.1287/mnsc.2019.00578 .

Solving Domain-Independent Dynamic Programming Problems with Anytime Heuristic Search

Domain-independent dynamic programming (DIDP) is a recently proposed model-based paradigm for combinatorial optimization where a problem is formulated as dynamic programming (DP) and solved by a generic solver. In this paper, we develop anytime heuristic search solvers for DIDP, which quickly find a feasible solution and continuously improve it to prove optimality. We implement six anytime heuristic search algorithms previously used as problem-specific methods and evaluate them on nine different problem classes. Our experimental results show that most of the anytime DIDP solvers outperform an existing A*-based solver, mixed-integer programming, and constraint programming in proving optimality, solution quality, and primal integral across multiple problem classes. In particular, complete anytime beam search (CABS) performs the best, improving on the best-known solution for one instance of traveling salesman problem with time windows and closing five instances of one-to-one multi-commodity pick-and-delivery traveling salesman problems.

Optimization of track distribution of industrial railway stations between car designations

Purpose. To improve the method for distributing the tracks of industrial railway stations for the accumulation of car groups between separate assignments. The optimization problem is to find such a distribution of classification work between the industrial marshalling yard and freight stations, as well as such a distribution of marshalling yard tracks between individual destinations, which ensures the minimum time expenditures on shunting work. Methodology. The studies were carried out using the methods of the theory of railway operation, simulation modeling and dynamic programming. Findings. The optimization problem of distribution of sorting work between the marshalling yard and freight stations of an industrial enterprise, as well as searching for such a distribution of marshalling yard tracks between individual destinations, which ensures the minimum time expenditures for shunting work, has been solved. Originality. The paper proposes a method for formalizing and solving the problem of distributing the tracks of industrial railway stations for the accumulation of car groups between individual destinations as a dynamic programming problem. Unlike the existing methods, where the number of tracks is considered as a constraint, in the proposed method, the number of tracks is an objective function argument of minimizing the duration of shunting operations, which improves the quality of the solutions obtained. Practical value. The method proposed in the paper, due to the rational distribution of the existing track arrangement of railway stations, makes it possible to reduce the time expenditures on the making- and breaking-up the trains and shunting transfers and, due to this, increase the carrying capacity of the stations, as well as reduce the cost of production of enterprises.

Distributed randomized multiagent policy iteration in reinforcement learning

We propose a distributed randomized policy iteration algorithm for infinite horizon dynamic programming problems for which the control at each stage is m-dimensional. The traditional policy iteration algorithm involves performing a minimization over an m-dimensional constraint set and has a computational complexity that increases exponentially in m, resulting in an intractable combinatorial search problem. In each iteration, our algorithm performs a series of sequential minimizations followed by policy evaluation and policy improvement using the policy that attains the minimum cost over the sequential minimizations. Our algorithm is well-suited for parallel computation, has a complexity that increases linearly in m, and converges to an agent-by-agent optimal policy. We characterize sufficient conditions for which our algorithm generates a globally optimal policy that coincides with that obtained from standard policy iteration.

Keep Your Distributed Data Warehouse Consistent at a Minimal Cost

Large data warehouses store interdependent tables that are updated independently in response to business logic changes or late arrival of critical data. To keep the warehouse consistent, changes to upstream tables need to be propagated to downstream tables in a timely fashion. However, a naive change propagation algorithm can cause many unnecessary updates or recalculations of downstream tables, which drives up the cost of data warehouse management. In this paper, we describe our solution that can ensure the eventual consistency of the data warehouse while avoiding unnecessary table updates. We also show that the optimal trade-off between computational cost reduction and meeting data freshness constraints can be found by solving a dynamic programming problem. The proposed solution is currently in production to manage the YouTube Data Warehouse and has reduced update requests by 25% by eliminating non-trivial duplicates. These requests would have been carried out by large batch jobs over big data. Eliminating them has led to a proportionate reduction in computing resources. One key advantage of our approach is that it can be used in a heterogeneous, distributed data warehouse environment where the operator software may not have complete control over the query processors. This is because our approach only relies on having dependency information for tables and can operate on the post-state of data sources.

An approximate dynamic programming approach for the maintenance optimisation of networked critical infrastructures

The continuous performance of networked infrastructures, such as transportation, telecommunications, and power transmission, is critical to the economic development and social well-being of society. To ensure their reliability in continuously meeting prescribed demand, cost-effective maintenance is essential. To this end, the influence of maintenance actions measures on the reliability of an infrastructure need to be considered both during their implementation and after their completion. To address this problem and to consider the uncertainty of future states, we model the multi-period optimisation of an infrastructure’s maintenance planning as a stochastic dynamic programming problem. An approximate dynamic programming approach is developed to deal with the computational complexity and solve the described problem. The proposed approach can help deal with the uncertainty of future infrastructure states by determining the optimal maintenance plan for each possible state in advance.

Addressing an open problem of Choudhury et al. “Fuzzy Sets Syst. 8 (2013), 84-97”

The question of relaxing the compatible hypothesis of the pair of mappings in fixed point theory has always been remained an open problem. We address such an open problem raised by Choudhury et al. [4] and also explicitly settles the issue of monotone and continuity hypotheses of the involved mappings in coupled coincidence point results. Moreover, we state a gap in an example given in [3] and repair it. Application to the dynamic programming problem shows the usability of present work. Finally, we also propose an open problem for further investigation.

Sensitivity-based dynamic performance assessment for model predictive control with Gaussian noise

Economic model predictive control and tracking model predictive control are two popular advanced process control strategies used in various of fields. Nevertheless, for a given process, which controller should be chosen to achieve better performance is uncertain when noise exists. To this end, a sensitivity-based performance assessment approach is proposed to pre-evaluate the dynamic economic and tracking performance of them and guide the controller selection in this work. First, their controller gains around the optimal steady state are evaluated using the sensitivities of corresponding constrained dynamic programming problems. Second, the controller gains are substituted into the control loop to derive the propagation of process and measurement noise. Subsequently, the Taylor expansion is introduced to simplify the calculation of variance and mean of each variable. Finally, the tracking and economic performance surfaces are plotted and the performance indices are precisely calculated through integrating the objective functions and the probability density functions. Moreover, boundary moving (i.e., back off) and target moving can be pre-configured to guarantee the stability of controlled processes based on the proposed approach. Extensive simulations under different cases prove that the proposed approach can provide useful guidance on performance assessment and controller design.

Meta-Learning Based Classification for Moving Object Trajectories in Mobile IoT

The proliferation and ubiquity of GPS-enabled mobile Internet of Things (IoT) devices (e.g., drones and unmanned robotic devices) across many disciplines has generated substantial interest in the analysis and mining of trajectory data for enhancing public services such as air pollution monitoring and road safety. Among all trajectory analysis techniques in mobile IoT, classifying moving device trajectories is a fundamental research problem due to its importance for numerous important mobile IoT applications such as travel demand analysis and animal migration patterns. However, existing classification methods for moving object trajectories either are dynamic programming problems (such as similarity-based methods with dynamic time wrapping or Fréchet distance) with a quadratic time complexity in all cases, or need abundant labeled samples to training classification model, thus limit the scalability of these methods for given resource-constrained mobile IoT devices. In this study, we propose a deep learning-based approach for trajectory classification in mobile IoT with linear time complexity by firstly encoding a trajectory as a vector via deep representation learning, then utilizing a meta-learning based approach to learn the ability of classifying trajectory with few annotated samples. Experiments on a massive trajectory dataset show that the proposed approach outperforms state-of-the-art baselines consistently and significantly.

Latency Optimization for Hybrid GEO–LEO Satellite-Assisted IoT Networks

Benefiting from the development of satellite onboard processing capability, the orbital computing can be realized by deploying edge computing servers on satellites to reduce the task processing latency. However, edge computing based on geostationary earth orbit (GEO) or low earth orbit (LEO) alone can hardly meet the latency requirements of satellite assisted internet of things (SIoT) services. Moreover, the uneven distribution of tasks generated by SIoT devices will also cause the load unbalancing among different satellites. In this paper, hybrid GEO-LEO SIoT networks is investigated with joint computing and communication resource allocation. To tackle the load unbalancing problem, tasks generated by SIoT devices can be processed by collaborative LEO satellites or forwarded to gateways on ground via GEO satellite. Thus, the joint task offloading, communication and computing resources allocation for the hybrid SIoT network can be formulated as a mixed integer dynamic programming problem with satellites-ground cooperation and inter-satellite cooperation via the inter-satellite links. Then, an intelligent task offloading and multi-dimensional resources allocation algorithm (TOMRA) is proposed to minimize the latency of task offloading and processing. Firstly, a method base on deep reinforcement learning is utilized to solve the subproblem of task offloading and channel allocation. And then, convex optimization is adopted to solve the sub-problem of computing resource allocation under fixed offloading and channel allocation decisions. Simulation results show that the proposed TOMRA can achieve better performance than the reference schemes.

Transmission of information in automated special purpose systems

This paper is devoted to solving the problem of optimizing the message transport service, which is one of the basic for automated special purpose systems. The NATO model of network-centric transformation of automated control and communication systems, which is presented in the form of a general plan for the development of information systems, was taken as the basis for the construction of special-purpose systems. An analysis of the development of information systems was carried out within the framework of the program of modernization and improvement of automated management systems. Under this program, large-scale work was carried out on the creation of new mobile nodes and means of communication for the perspective automated communication system at the tactical level Warfighter Information Network-Tactical. Therefore, it is proposed to use wireless, decentralized, mobile IP networks such as MANET (Mobile Ad hoc Network) for automated special purpose systems at the tactical level. Each of the mobile devices of such network can move independently in any direction, and, as a result, often break and establish connections with neighbors. For the MANET network the messaging service is considered in the context of: one automated workplace, one domain and interconnection between domains. The efficiency of information transmission systems is considered. Circular transmission algorithms for messages with the highest urgency category are defined and a control algorithm is provided for selecting the optimal message transmission channel in which the time spent on message transmission between these subscribers is minimal (dynamic programming problem with additive effect function). Bellman's functional equation for the conditional optimization stage was adapted for this problem.

Costly multi-unit search

We formulate and solve a costly multi-unit search problem for the optimal selling of a stock of goods. Our showcase application is an inventory liquidation problem with holding costs, such as warehousing, salaries or floor planning. A seller faces a stream of buyers periodically arriving with random capped demands. At each decision point, he decides how to price each unit and also whether to stop searching or not. We set this as a dynamic programming problem and solve it inductively by characterizing optimal search rules and reservation prices. Our inductive logic does not rely in the use of the first order conditions for optimal pricing and optimal search decisions.We show that combining multiple units with a fixed per period search cost might translate into non-monotone selling costs and reservation prices. This lack of monotonicity naturally leads to discontinuities of the pricing strategy. In particular, the seller optimally employs strategies such as bundling, and more sophisticated ones that endogenously combine purchase premiums, when inventory is large, with clearance sales and discounts, when inventory is low. Instead, a variable search cost always yields an increasing marginal selling cost and thus can only make sense of pricing strategies that employ purchase premiums.Our model extends search theory by explicitly accounting for the effects of search costs on optimal multi-unit pricing strategies, pushing it into a richer class of problems and offering solutions that extend beyond optimal stopping rules.

Optimization of the offshore wind power grid-connected structure based on an improved genetic algorithm

Large-scale offshore wind power generation has become one of the research hotspots in the development of new energy in the world. However, the planning of far-reaching offshore wind power is faced with many technical difficulties, such as the need to consider the optimization of line transmission capacity caused by the truncation effect in combination with the wind power output probability, and solve the problem that traditional algorithms cannot solve the dynamic programming problem, and the greedy algorithm for solving the dynamic programming problem cannot solve the global optimal solution. Therefore, in order to solve the above problems, based on the full life cycle (FLC) investment calculation of the collecting and transmission system, the topology optimization model of the offshore wind power system is established. In addition, in order to solve the grid-connection problem of offshore wind farms, an improved topology optimization algorithm based on greedy idea is proposed. The optimization results show that the critical values of economic transmission distance of HVAC, FFTS and HVDC are usually 50 km and 170 km respectively. Finally, the collection scheme of the wind farm cluster including three far-reaching offshore wind farms are analyzed, and the results show that the total cost of the optimized point-to-point scheme is reduced by 5.6% and 10.2% respectively compared with the star topology and the circular collection topology, with significant economic benefits.

Economic model predictive control with terminal set dynamic programming for tracking control

AbstractThis paper focuses on vehicle tracking control issues with consideration of energy efficiency, ride comfort and tracking performance. To address this issue, an economic model predictive control (EMPC) framework is proposed by taking average performance into account and introducing a time‐varying terminal set. We integrate an economic model predictive controller and a terminal controller with a feedback gain related to the time‐varying terminal set into the proposed EMPC framework. An acceleration‐dependent average constraint is designed and incorporated into the MPC optimization problem to facilitate the convergence of the actual acceleration to the desired one, which enables the vehicle to implement a smooth driving style that boosts energy economy. A tuning factor in the average constraint has the capacity to balance energy consumption and control performance. Under the premise of invariance constraints, a dynamic terminal set is constructed by solving a dynamic programming problem online, which enables an arbitrary location and scale of the terminal set at each iteration by taking the current and predicted tracking errors into account. This allows a reduction in the conservativeness of an economic model predictive controller in the sense of both the region of attraction and the cost bound. Recursive feasibility of the MPC optimization problem and the terminal set dynamic programming is ensured, and average performance is not worse than the performance with an optimal steady‐state operation. Moreover, the convergence of the closed‐loop tracking error to the optimal steady‐state is guaranteed by using the dissipativity theory. The effectiveness of the proposed algorithm is verified by numerical comparisons with different controller parameters and standard EMPC.

Valuation of an unexplored oilfield under uncertain oil price and reservoir condition: A stochastic dynamic programming approach with simulation-based reward function

We develop an unexplored oilfield valuation model under uncertain exploration outcomes, reservoir conditions, and oil prices. The exploration outcomes follow a binomial process, while oil prices are represented using the Schwartz-Smith model. The reservoir conditions are characterized by the joint probability distribution of post-discovery parameters estimated using a machine learning approach. The corresponding valuation model is a typical stochastic dynamic programming problem with a simulation-based reward function. The net cash flow lattice takes the form of a recombining quadrinomial derived from the discrete representation of the Schwartz-Smith oil price model. We apply backward induction with embedded real-coded genetic algorithms to the net cash flow lattice to calculate the oilfield value. The model allows for field abandonment before lease expiration if the remaining reserve is uneconomical. To improve computational efficiency, we combine the Latin hypercube sampling and antithetic variates to reduce the variances. The model is implemented in an unexplored oilfield under two scenarios of oil presence probability: 100% and 75%. In the first scenario, we obtained a mean oilfield value of US$5.5 million with a CVaR of US$0.79 million, while in the second, we came up with a mean of -US$0.77 million and a CVaR of US$31.74 million.