Dynamic Programming Problem Research Articles

Perov's contraction principle and dynamic programming with stochastic discounting

This paper shows the usefulness of Perov's contraction principle, which generalizes Banach's contraction principle to a vector-valued metric, for studying dynamic programming problems in which the discount factor can be stochastic. The discounting condition β<1 is replaced by ρ(B)<1, where B is an appropriate nonnegative matrix and ρ denotes the spectral radius. Blackwell's sufficient condition is also generalized in this setting. Applications to asset pricing and optimal savings are discussed.

Dynamic expansion of flexible capacity with and without pricing coordination

This research presents an optimal multi-period dynamic capacity expansion framework for a shared flexible capacity producing two products. The framework allows modeling coordinated and non-coordinated capacity and marketing decisions under demand uncertainties. In our two-level decision process, the upper level considers the strategic capacity expansion and formulates multi-period uncertainty as a stochastic dynamic programming (DP) problem, and the lower level considers each period’s pricing and capacity allocation decisions in response to available capacity and market demand. We compare capacity expansion decisions under centralized pricing schemes that coordinate the pricing decisions of different products based on the overall interest of the firm and the decentralized pricing schemes based on the interests of individual product managers. Numerical results show that adopting a decision coordination strategy may help firms to maintain flexibility and reduce capacity expansion investment while enhancing overall profitability.

Searching using B/B+ Tree in Database Management System

This is a research-based project and the basic point motivating this project is learning and implementing algorithms that reduce time and space complexity. In the first part of the project, we reduce the time taken to search a given record by using a B/B+ tree rather than indexing and traditional sequential access. It is concluded that disk-access times are much slower than main memory access times. Typical seek times and rotational delays are of the order of 5 to 6 milliseconds and typical data transfer rates are of the range of 5 to 10 million bytes per second and therefore, main memory access times are likely to be at least 4 or 5 orders of magnitude faster than disk access on any given system. Therefore, the objective is to minimize the number of disk accesses, and thus, this project is concerned with techniques for achieving that objective i.e. techniques for arranging the data on a disk so that any required piece of data, say some specific record, can be located in a few I/O’s as possible. In the second part of the project, Dynamic Programming problems were solved with Recursion, Recursion With Storage, Iteration with Storage, Iteration with Smaller Storage. The problems which have been solved in these 4 variations are Fibonacci, Count Maze Path, Count Board Path, and Longest Common Subsequence. All 4 variations are an improvement over one another and thus time and space complexity are reduced significantly as we go from Recursion to Iteration with Smaller Storage.

Routing policy choice prediction in a stochastic network: Recursive model and solution algorithm

We propose a Recursive Logit (STD-RL) model for routing policy choice in a stochastic time-dependent (STD) network, where a routing policy is a mapping from states to actions on which link to take next, and a state is defined by node, time and information. A routing policy encapsulates travelers’ adaptation to revealed traffic conditions when making route choices. The STD-RL model circumvents choice set generation, a procedure with known issues related to estimation and prediction. In a given state, travelers make their link choice maximizing the sum of the utility of the outgoing link and the expected maximum utility until the destination (a.k.a. value function that is a solution to a dynamic programming problem). Existing recursive route choice models and the corresponding solution approaches are based on the assumption that network attributes are deterministic. Hence, they cannot be applied to stochastic networks which are the focus of this paper.We propose an efficient algorithm for solving the value function and its gradient, critical for parameter estimation. It is based on partitioning the state space and decomposing costly matrix operations into a series of simpler ones. We present numerical results using a synthetic network and a network in Stockholm, Sweden. The estimation running time has a 20-30 times speed-up due to matrix decomposition. The estimated model parameters have realistic interpretations. Specifically, travelers are more likely to be adaptive to realized travel times during a longer trip, and more sensitive to travel time when travel time variability is higher. The STD-RL model performs better in predicting route choices than the RL model in a corresponding static and deterministic network.

Low-carbon technology development under multiple adoption risks

Reducing greenhouse gas emissions is now an important global issue. The promotion and application of low-carbon technologies is one important method of achieving greenhouse gas reduction. The core problem is identifying the types of technology to introduce, when and to what degree, while satisfying the related sustainable goals. This study proposes a frame of development path design for low-carbon technology. Contrary to existing methods, our innovation lies in the introduction of multiple uncertainties in the technology adoption process. We consider four main types of technical risks, and provide corresponding quantitative descriptions. Our framework describes a class of technology development path problems as dynamic programming problems with different random constraints. Using existing numerical simulation methods, we obtain solutions of the original problem at a certain level of confidence, and discuss important issues, such as the learned robust features, and the relevant sensitivity of the model in a given solution space. Related results confirm the effectiveness of our model. Notably, our analysis framework can be generalized to solve similar problems, especially, in the characterization of technical risks.

Power and Traffic Nexus: From Perspective of Power Transmission Network and Electrified Highway Network

The increasing prevalence of electric vehicles (EVs) and emerging dynamic wireless charging techniques have increased the interdependence between the power and transportation systems. This article investigates a new nexus scenario-the electrified highway network (EHN) and power transmission network (PTN). The independent but interrelated operation of the two networks is analyzed. On the EHN side, a combined charging-driving navigation (CCDN) model, which rigorously considers EV travel speed and charge/discharge behaviors, is proposed and formulated as an irregular dynamic programming problem. A chronological search algorithm is designed to derive the optimal charging-driving decision sequences. The economic operation of the PTN is formulated as a direct current optimal power flow problem. Hourly location marginal prices (LMPs) and charge/discharge demands are exchanged between the two networks. The abovementioned LMP-based interaction forms a Nash-type game, in which both networks aim to minimize their own operation costs. A best-response decomposition algorithm combined with a continuous LMP method is developed to identify the equilibrium state. Numerical results demonstrate the effectiveness of the CCDN model in modeling EV charging-driving behaviors in the EHN. Moreover, the results show that, under certain conditions, mobility of EVs along the EHN can economically relieve the transmission congestion and reduce the operation costs of both networks.

Commentary - Methods to find all the edges on any of the shortest paths between two given nodes of a directed acyclic graph

This article puts forth all the existing methods proposed by the various authors of the Stack Exchange community to find all the edges on any shortest path between two given nodes of a directed acyclic graph. For a directed acyclic graph with N number of nodes, an exponential number of paths are possible between any two given nodes and, thus, it is not feasible to compute every path and find the shortest ones in polynomial time to generate a set of all edges that contribute or make any of the shortest paths. The methods discussed in this article are not limited only to this specific use case, but have a much broader scope in graph theory, dynamic programming and counting problems. Generally, various other questions and answers, raised on the community portal having similar scope to those that the users specifically seek, do not receive sufficient hits and, hence, enough attention and votes for various reasons worth contemplating. Therefore, this article also aims to highlight the various scopes of the methods discussed in this article and acknowledge the efforts of the authors, moderators and contributors of the Stack Exchange community for their expertise and time to write precise answers and share their opinions and advice. Finally, it also appeals to all the other beneficiaries in the community to use their privileges responsibly and upvote the posted answers, if they helped solve their queries, as one upvote is free of cost.

Coordinated Optimal Voyage Planning and Energy Management of All-Electric Ship With Hybrid Energy Storage System

All-electric (AES) ship power system (SPS) generally employs energy storage (ESS) to improve operation efficiency, redundancy, and flexibility while reducing environmental impacts. Depending on the operating characteristic, ramp rate, and load variation of the SPS, single or hybrid energy storage systems (HESS) with different operating characteristics are utilized to prevent frequent cycling, high depth of discharge (DOD), and accelerated degradation. Hence, this research work focuses on the coordination of optimal voyage, and multi-objective energy management of AES with HESS to optimize the vessel route, operation cost, emission, and degradation of ESS. The optimal route planning is carried out as a dynamic programming problem based on the sea states, and augmented ϵ-constraint and reactive approach to energy management strategy are adopted to solve the energy dispatch of the AES while addressing the uncertainty present in the load forecasting. The simulation results indicate that the voyage planning with the consideration of the sea states has strong influences on the path taken and velocity of the AES which in term will influence the propulsion power requirement. It also indicates that the operating characteristic of the HESS and uncertainty present in the load forecasting should be carefully addressed to achieve a realistic operation schedule.

Capacity management of CT department with service time differences and emergency nonpreemptive priority

In the presence of different service times of regular patients and nonpreemptive priority of emergency patients at the same day, this paper considers an advance appointment-scheduling problem of medical resources in which a finite amount of same-day service capacity should be allocated to different patients. This booking problem is modeled as a dynamic programming (DP) problem. The structural properties of DP model analytically reveal a booking limit policy. The booking limit numbers of regular patients do not monotonically increase with their revenues because of the service time differences. A numerical analysis is provided to test the performance of our booking limit policy by comparing polices in previous studies. Simulation results show that our policy provides a positive improvement in terms of the total reward of the service system, even when patients do not have perfect adherence to appointments (e.g., patient no-shows and unpunctuality). The numerical experiments demonstrate that the dedicated medical resource cannot improve emergency patients’ access to service but deteriorated their direct waiting time.

Dynamic Transitional Droops for Seamless Line-Switching in Islanded Microgrids

Line-switching operations introduce several concerns to microgrid operators since they trigger reliability and power quality issues in addition to increasing the wear and tear cost of switching devices. This paper proposes a novel methodology for achieving seamless line-switching in low-inertia islanded microgrids. The proposed methodology aims at mitigating the detrimental line-switching impacts by minimizing the power flow in the switched lines at the moments of executing the switching operations. The minimization of the power flow is realized by optimizing the droop control parameters of the DGs during the transition from one network topology to another. Besides, during executing a series of line-switching operations, the sequence of switching is also optimized to minimize the total switched power. While optimizing the line-switching execution sequence, the transitional droops are optimized for each line-switching operation within that sequence. A dynamic mixed-integer nonlinear programming problem is formulated to find both the optimized dynamic transitional droops and line-switching sequence simultaneously. Simulation results demonstrate the effectiveness of the proposed seamless line-switching methodology where the switched power can be reduced by more than 80% for a single line-switching operation and 54% considering a series of four switch exchanges in addition to achieving several instances of zero-current switching. As a result, a significant decrease in the peak of voltage transients has been recorded for some cases.

A General Framework for Bounding Approximate Dynamic Programming Schemes

For years, there has been interest in approximation methods for solving dynamic programming problems, because of the inherent complexity in computing optimal solutions characterized by Bellman's principle of optimality. A wide range of approximate dynamic programming (ADP) methods now exists. It is of great interest to guarantee that the performance of an ADP scheme be at least some known fraction, say β, of optimal. This letter introduces a general approach to bounding the performance of ADP methods, in this sense, in the stochastic setting. The approach is based on new results for bounding greedy solutions in string optimization problems, where one has to choose a string (ordered set) of actions to maximize an objective function. This bounding technique is inspired by submodularity theory, but submodularity is not required for establishing bounds. Instead, the bounding is based on quantifying certain notions of curvature of string functions; the smaller the curvatures the better the bound. The key insight is that any ADP scheme is a greedy scheme for some surrogate string objective function that coincides in its optimal solution and value with those of the original optimal control problem. The ADP scheme then yields to the bounding technique mentioned above, and the curvatures of the surrogate objective determine the value β of the bound. The surrogate objective and its curvatures depend on the specific ADP.

Accelerating DNA pairwise sequence alignment using FPGA and a customized convolutional neural network

An optimized software and hardware digital implementation of two widely used DNA sequence alignment algorithms based on lookup table(LUT) is illustrated in this study. These algorithms are the best means for identifying similar regions between sequences. The proposed implementation relies on the complete parallelization of these foundational algorithms under certain limitations to overcome most of the problems of dynamic programming and hardware implementation. The proposed method takes O(N/4) calculation steps, where N is the length of each sequence with a minimum value of four (i.e., N = 4,8,12,…). A performance comparison between the state of art and our proposed algorithm is conducted for software and hardware implementation. Combinational circuits are used for FPGA-based hardware implementation of DNA sequence alignment algorithms. Performance and device resource usage are evaluated for different hardware designs. A customized convolution neural network model is used to implement global alignment and achieve 98.3% accuracy.

The uncoupled microbial fed-batch fermentation optimization based on state-dependent switched system

In the actual microbial fermentation process, excessive or insufficient substrate can produce inhibitory effects on cells growth. The artificial substrate feeding rules by past experiences have great blindness to keep substrate concentration in a given appropriate range. This paper considers that alkali feed depends on pH value of the solution and glycerol feed depends on glycerol concentration of the solution in the uncoupled microbial fed-batch fermentation process, and establishes a state-dependent switched system in which the flow rates of glycerol and alkali, the number of mode switches, the mode sequence and the switching times are prior unknown. To maximize the yield of target product 1,3-Propanediol (1,3-PD), we formulate a switching optimal control problem with the flow rates of glycerol and alkali, the number of mode switches, the mode sequence and the switching times as decision variables, which is a mixed-integer dynamic programming problem. For solving the mixed-integer dynamic programming problem, the control parametrization technique, the time scaling transformation and the embedded system technology are used to obtain an approximate parameter optimization problem. By using a parallel optimization algorithm, we obtain the optimal control strategies. Under the obtained optimal control strategies, the 1,3-PD yield at the terminal time is increased significantly compared with the previous results.

Optimal structure of joint inventory-pricing management with dual suppliers and different lead times

We consider a joint inventory-pricing control problem in a single-product, periodic-review, dual-supplier inventory system. The two suppliers have different lead times. One expedited supplier offers instantaneous replenishment, and one regular supplier requires an L-period lead time for delivery. The supply quantity is stochastic and the demand is price-dependent. For the expedited inventory replenishment, we characterize the optimal policy as a state-dependent almost-threshold policy by extending the stochastically linear in mid-point to a multidimensional setting. To investigate the optimal regular inventory replenishment and pricing policy, we propose the notions of partially stochastic translation (PST) and increasing partially stochastic translation (IPST), which help in obtaining the antimultimodularity preservation in dynamic programming problems. We provide properties, sufficient conditions, and examples for PST and IPST functions. By applying PST and IPST, we obtain the antimultimodularity of the profit functions. The antimultimodular profit functions ensure that the optimal regular ordering quantity and the optimal price are monotone in the current inventory level and outstanding order quantities. Moreover, we reveal that as the time interval increases, the effects of previous outstanding orders on the optimal regular ordering and pricing decisions are decreasing and increasing, respectively. PST and IPST also enable us to further characterize the optimal expedited ordering quantity as decreasing in the inventory level. However, the optimal expedited ordering quantity can be non-monotone with respect to the outstanding order quantities, as shown in the example.

Blind separation of noncooperative paired carrier multiple access signals based on improved quantum‐inspired evolutionary algorithm and receding horizon optimization

AbstractThe single‐channel blind source separation of paired carrier multiple access (PCMA) signal is a key technology in satellite communications. Due to the high‐order complexity of existing separation methods and the uncertainty of channel parameters, the processing of noncooperative PCMA signals with long memory remains a great challenge. In this article, the blind separation was solved as the joint channel estimation and sequence detection, and a novel blind separation algorithm based on the improved quantum‐inspired evolutionary algorithm (IQEA) and receding horizon optimization was proposed. Considering the practical scenarios, a mixed PCMA signal model with noninteger period sampling was presented in advance. Combined with the proposed signal model, the IQEA was applied to recover the uplink signals from the mixed PCMA signals by its iterative estimation. Meanwhile, to reduce the computational complexity caused by long memory of PCMA signals, the receding horizon optimization was introduced to process the whole signal sequence as a dynamic programming problem. The simulation results verified that the proposed RH‐IQEA outperforms existing state‐of‐the‐art separation algorithms in the blind separation of noncooperative PCMA signals, with higher accuracy, lower‐order complexity and stronger robustness to parameter estimation errors.

INVESTMENT IN INNOVATIVE DEVELOPMENT OF COMPETITIVENESS OF RESOURCE AND PRODUCTION POTENTIAL OF MINING ENTERPRISES

The subject of the research is the processes of ensuring the competitiveness of the potential of mining enterprises based on certain key aspects of methodology and practice of rational management of investment distribution, which are inherent in the investment and innovation activities of subsidiaries within the parent company. The purpose and objectives of the study is to improve theoretical and methodological approaches to increase the competitiveness of the potential of mining enterprises on the basis of rational management of the allocation of investment resources in the process of their innovative development. Method (methodology). Solving issues of rational allocation of investment resources from the standpoint of proper competitiveness of mining enterprises is proposed to be carried out according to the method of optimizing multi-stage processes - the problem of dynamic programming of resource allocation between enterprises based on the principle of optimality R. Bellman. Results. As a result of research it is established that the most important criterion for optimizing the program of innovative development on the basis of rational distribution of investment resources is to choose the cost of sales, due to the dependence of financial performance of mining companies on total costs. The implementation of the proposed method of optimizing the allocation of investment resources is considered on the example of jointly operating in one company mining companies. Conclusions. Based on the generalization of research on theoretical and practical aspects related to improving the efficiency of managing the competitiveness of resource and production potential of mining enterprises, a methodological approach to prioritizing the allocation of investment resources in the context of innovation development working together in one company. The application of the developed methodological approach in practice allows the top management of large vertically integrated companies to rationally direct the allocation of investment resources to the stable innovative development of subsidiaries.

Costly Multi-Unit Search

We formulate and solve a costly multi-unit search problem for the optimal selling of a stock of goods. Our showcase application is an inventory liquidation problem with fixed holding costs, such as warehousing, salaries or floor planning. A seller faces a stream of buyers periodically arriving with random capped demands. At each decision point, he decides how to price each unit and also whether to stop search or not. We set this as a dynamic programming problem and solve it inductively by characterizing optimal search rules and reservation prices. We show that combining multiple units with a fixed per period search cost might translate into non-monotone selling costs and reservation prices. This lack of monotonicity naturally leads to discontinuities of the pricing strategy. In particular, the seller optimally employs strategies such as bundling, and more sophisticated ones that endogenously combine purchase premiums, when inventory is large, with clearance sales and discounts, when inventory is low. Our model extends search theory by explicitly accounting for the effects of fixed costs on optimal multi-unit pricing strategies, pushing it into a richer class of problems and offering solutions that extend beyond optimal stopping rules.

Exploiting Symmetry in High-Dimensional Dynamic Programming

We propose a new method for solving high-dimensional dynamic programming problems and recursive competitive equilibria with a large (but finite) number of heterogeneous agents using deep learning. The curse of dimensionality is avoided due to four complementary techniques: (1) exploiting symmetry in the approximate law of motion and the value function; (2) constructing a concentration of measure to calculate high-dimensional expectations using a single Monte Carlo draw from the distribution of idiosyncratic shocks; (3) sampling methods to ensure the model fits along manifolds of interest; and (4) selecting the most generalizable over-parameterized deep learning approximation without calculating the stationary distribution or applying a transversality condition. As an application, we solve a global solution of a multi-firm version of the classic Lucas and Prescott (1971) model of investment under uncertainty. First, we compare the solution against a linear-quadratic Gaussian version for validation and benchmarking. Next, we solve nonlinear versions with aggregate shocks. Finally, we describe how our approach applies to a large class of models in economics. Institutional subscribers to the NBER working paper series, and residents of developing countries may download this paper without additional charge at www.nber.org.

Impact of COVID-19 Type Events on the Economy and Climate Under the Stochastic DICE Model

The classical DICE model is a widely accepted integrated assessment model for the joint modeling of economic and climate systems, where all model state variables evolve over time deterministically. We reformulate and solve the DICE model as an optimal control dynamic programming problem with six state variables (related to the carbon concentration, temperature, and economic capital) evolving over time deterministically and affected by two controls (carbon emission mitigation rate and consumption). We then extend the model by adding a discrete stochastic shock variable to model the economy in the stressed and normal regimes as a jump process caused by events such as the COVID-19 pandemic. These shocks reduce the world gross output leading to a reduction in both the world net output and carbon emission. The extended model is solved under several scenarios as an optimal stochastic control problem, assuming that the shock events occur randomly on average once every 100 years and last for 5 years. The results show that, if the world gross output recovers in full after each event, the impact of the COVID-19 events on the temperature and carbon concentration will be immaterial even in the case of a conservative 10% drop in the annual gross output over a 5-year period. The impact becomes noticeable, although still extremely small (long-term temperature drops by 0.1°C), in a presence of persistent shocks of a 5% output drop propagating to the subsequent time periods through the recursively reduced productivity. If the deterministic DICE model policy is applied in a presence of stochastic shocks (i.e. when this policy is suboptimal), then the drop in temperature is larger (approximately 0.25°C), that is, the lower economic activities owing to shocks imply that more ambitious mitigation targets are now feasible at lower costs.

Exploiting Symmetry in High-Dimensional Dynamic Programming

We propose a new method for solving high-dimensional dynamic programming problems and recursive competitive equilibria with a large (but finite) number of heterogeneous agents using deep learning. The curse of dimensionality is avoided due to four complementary techniques: (1) exploiting symmetry in the approximate law of motion and the value function; (2) constructing a concentration of measure to calculate high-dimensional expectations using a single Monte Carlo draw from the distribution of idiosyncratic shocks; (3) sampling methods to ensure the model fits along manifolds of interest; and (4) selecting the most generalizable over-parameterized deep learning approximation without calculating the stationary distribution or applying a transversality condition. As an application, we solve a global solution of a multi-firm version of the classic Lucas and Prescott (1971) model of investment under uncertainty. First, we compare the solution against a linear-quadratic Gaussian version for validation and benchmarking. Next, we solve nonlinear versions with aggregate shocks. Finally, we describe how our approach applies to a large class of models in economics.