Dynamic Programming Recursion Research Articles

Cost-aware defense for parallel server systems against reliability and security failures

Parallel server systems in transportation, manufacturing, and computing heavily rely on dynamic routing using connected cyber components for computation and communication. Yet, these components remain vulnerable to random malfunctions and malicious attacks, motivating the need for resilient dynamic routing that is both traffic-stabilizing and cost-efficient. In this paper, we consider a parallel server system with dynamic routing subject to reliability and stability failures. For the reliability setting, we consider an infinite-horizon Markov decision process where the system operator strategically activates a protection mechanism upon each job arrival based on traffic state observations. We prove that an optimal deterministic threshold protecting policy exists based on the dynamic programming recursion of the Hamilton–Jacobi–Bellman equation. For the security setting, we extend the model to an infinite-horizon stochastic game where the attacker strategically manipulates routing assignments. We show that both players follow a threshold strategy at every Markov perfect equilibrium. For both failure settings, we also analyze the stability of the traffic queues. Finally, we develop approximate dynamic programming algorithms to compute the optimal/equilibrium policies and present numerical examples/experiments for validation and illustration.

Synchronized Deliveries with a Bike and a Self-Driving Robot

Online e-commerce giants are continuously investigating innovative ways to improve their practices in last-mile deliveries. Inspired by the current practices at JD.com (the largest online retailer by revenue in China), we investigate a delivery problem that we call the traveling salesman problem with bike and robot (TSPBR), where a cargo bike is aided by a self-driving robot to deliver parcels to customers in urban areas. We present two mixed-integer linear programming models and describe a set of valid inequalities to strengthen their linear relaxation. We show that these models can yield optimal solutions of TSPBR instances with up to 60 nodes. To efficiently find heuristic solutions, we also present a genetic algorithm based on a dynamic programming recursion that efficiently explores large neighborhoods. We computationally assess this genetic algorithm on instances provided by JD.com and show that high-quality solutions can be found in a few minutes of computing time. Finally, we provide some managerial insights to assess the impact of deploying the bike-and-robot tandem to deliver parcels in the TSPBR setting.

Mono-valent salt corrections for RNA secondary structures in the ViennaRNA package

BackgroundRNA features a highly negatively charged phosphate backbone that attracts a cloud of counter-ions that reduce the electrostatic repulsion in a concentration dependent manner. Ion concentrations thus have a large influence on folding and stability of RNA structures. Despite their well-documented effects, salt effects are not handled consistently by currently available secondary structure prediction algorithms. Combining Debye-Hückel potentials for line charges and Manning’s counter-ion condensation theory, Einert et al. (Biophys J 100: 2745-2753, 2011) modeled the energetic contributions of monovalent cations on loops and helices.ResultsThe model of Einert et al. is adapted to match the structure of the dynamic programming recursion of RNA secondary structure prediction algorithms. An empirical term describing the salt dependence of the duplex initiation energy is added to improve co-folding predictions for two or more RNA strands. The slightly modified model is implemented in the ViennaRNA package in such way that only the energy parameters but not the algorithmic structure is affected. A comparison with data from the literature show that predicted free energies and melting temperatures are in reasonable agreement with experiments.ConclusionThe new feature in the ViennaRNA package makes it possible to study effects of salt concentrations on RNA folding in a systematic manner. Strictly speaking, the model pertains only to mono-valent cations, and thus covers the most important parameter, i.e., the NaCl concentration. It remains a question for future research to what extent unspecific effects of bi- and tri-valent cations can be approximated in a similar manner.AvailabilityCorrections for the concentration of monovalent cations are available in the ViennaRNA package starting from version 2.6.0.

Throughput maximization and reliable wireless communication in NOMA using chained fog structure and weighted energy efficiency power allocation approach

Novel radio access strategies must be developed to support an unprecedented number of connected devices to increase radio spectral efficiency of 5G and outside, and non-orthogonal multiple features (NOMA) arisen as a possible contender. To support cognitive NOMA networks, power allocation and NOMA-secondary user allocation is the effective technique for enhancing the resource utilization proficiency in power and spectrum domain. NOMA approach provides throughput enhancement to fulfil the demands of next group of Wireless Communication Networks. In this manuscript, a Chained Fog Structure (CFS) with Weighted Energy Efficiency Power Allocation (WEE-PA) method is proposed for throughput maximization and reliable wireless communication (CFS-WEE-NOMA). The aim of this study is “to enhance the throughput by involving underlay NOMA with Chained Fog Structure and WEE-PA method”. Here, the proposed WEE-PA approach is applied to multicarrier NOMA with chained fog structure for enhancing the throughput. During this analysis, the computational complexity due to the consideration of high users and different subcarrier values are solved by the proposed CFS-WEE-NOMA approach. Thus, the proper user pairing, power allocation, and bandwidth sharing of the proposed methodology ensure seamless communication. The simulation of this work is done in MATLAB and the performance metrics like throughput, computational complexity, power consumption, energy efficiency, delay, sum rate is analysed. The proposed CFS-WEE-NOMA approach has achieved 8.6%, 7.7%, and 6.7% high throughput based on transmitted power, 13.6%, 8.6%, and 11.7% high throughput based on subcarrier, and 13.6%, 9.6%, and 12.7% high energy efficiency compared with the existing methods, like Dynamic Network Resource Allocation (DNRA) algorithm (DNRA-NOMA), Multiple Carrier Cell-Less Non-Orthogonal Multiple Access (MC-CL-NOMA), dynamic programming (DP) recursion framework based multicarrier NOMA systems (DPRF-MC-NOMA) methods respectively.

Numerical Solution of Dynamic Quantile Models

This paper studies dynamic programming for quantile preference models, in which the agent maximizes the stream of the future τ-quantile utilities, for τ∈(0,1). We suggest numerical methods, based on value function iterations, for solving the quantile recursive dynamic programming, and computing value and policy functions. In addition, we extend theoretical results to allow the dynamic quantile model to have a finite-horizon, instead of infinite-horizon. To illustrate the methods, we use an intertemporal consumption quantile model that has an explicit closed form solution for both the value and policy functions. Based on this example, we assess the accuracy of the numerical methods by computing and comparing theoretical and numerical value and policy functions, for several combinations of the parameters – discount factor, elasticity of intertemporal substitution, and risk attitude, which is measured by the quantile. Results document evidence that the suggested algorithm provides numerical solutions that are very close to theoretical counterparts, and also illustrate the usefulness and practicality of the proposed methods.

A polyhedral approach to least cost influence maximization in social networks

The least cost influence maximization problem aims to determine minimum cost of partial (e.g., monetary) incentives initially given to the influential spreaders on a social network, so that these early adopters exert influence toward their neighbors and prompt influence propagation to reach a desired penetration rate by the end of cascading processes. We first conduct polyhedral analysis on a substructure that describes influence propagation assuming influence weights are unequal, linear and additively separable. Two classes of facet-defining inequalities based on a mixed 0–1 knapsack set contained in this substructure are proposed. We characterize another exponential class of valid and facet-defining inequalities utilizing the concept of minimum influencing subset. We show that these inequalities can be separated in polynomial time efficiently. Furthermore, a polynomial-time dynamic programming recursion is presented to solve this problem on a simple cycle graph. For arbitrary graphs, we propose a new exponential class of valid inequalities that dominates the cycle elimination constraints and an efficient separation algorithm for them. A compact convex hull description for a special case is presented. We illustrate the effectiveness of these inequalities via a delayed cut generation algorithm in the computational experiments.

Robust Inventory Management: A Cycle-Based Approach

Problem definition: We study the robust formulation of an inventory model with positive fixed ordering costs, where the unfulfilled demand is either backlogged or lost, the lead time is allowed to be positive, the demand is potentially intertemporally correlated, and the information about the demand distribution is limited. Methodology/results: We propose a robust cycle-based policy that manages inventory by dividing the planning horizon into nonoverlapping inventory cycles, where an order is placed at the beginning of each cycle. Our policy selects the lengths and order quantities for all inventory cycles to minimize the worst-case total cost incurred over the planning horizon. When the uncertain demand belongs to a general polyhedral uncertainty set, the decisions in our policy can be computed by solving linear programs (LPs) for the backlogging model with any lead time and the lost-sales model with zero lead time; however, the number of LPs that need to be solved grows exponentially in the length of the planning horizon. In the special case where the uncertain demand belongs to a box uncertainty set, the decisions in our policy can be computed using a dynamic programming (DP) recursion whose complexity grows polynomially in the length of the planning horizon. We also propose a one-cycle look-ahead heuristic to handle large problem instances with a general polyhedral uncertainty set. This heuristic can be applied for both the backlogging and lost-sales models with any lead time, and it only requires solving LPs whose number grows quadratically in the length of the planning horizon. Results from extensive computational experiments clearly show that both a rolling-cycle implementation of our policy and the one-cycle look-ahead heuristic have very strong empirical performance. Managerial implications: Our robust cycle-based policy and the one-cycle look-ahead heuristic are conceptually simple and can accommodate multiple realistic features in inventory management problems. They provide a very effective approach to robust inventory management, especially in the lost-sales setting. Funding: Y. Chen was supported by a start-up grant from Nanyang Technological University. C. Wang was supported by the National Natural Science Foundation of China [Grant 71802115] and the Tsinghua University Initiative Scientific Research Program. Supplemental Material: The online appendix is available at https://doi.org/10.1287/msom.2022.1168 .

An exact approach for the constrained two-dimensional guillotine cutting problem with defects

This paper studies the constrained two-dimensional guillotine cutting problem with defects, whose objective is to cut a subset of given items from a defective sheet such that the profit of selected items is maximised. The guillotine cut constraint, which requires each cut must go through one side of the sheet to the opposite side, is considered. We solve this problem via a recursive dynamic programming approach. A set of upper bounds is proposed to keep the promising nodes. The normal points and raster points are extended to reduce the number of vertical and horizontal cuts by considering the effect of the defect. The experiment results show that our approach can solve most of the instances in the literature and outperforms existing approaches.

Nonagonal Neutrosophic Number and its Application in Optimization Technique

This article discusses Nonagonal Neutrosophic number and m-valued Nonagonal Neutrosophic number. The score function, the accuracy function, hamming distance, normalized hamming distance, Euclidean distance and normalized Euclidean distance of Nonagonal and m-polar Nonagonal Neutrosophic number are derived. Some de-neutrosophication method for Nonagonal Neutrosophic number and some properties of m-valued Nonagonal Neutrosophic number are proved. In this article the optimal path of an acyclic network is estimated using Neutrosophic α-cut grade, Neutrosophic Euclidean grade technique and dynamic programming recursion method through Nonagonal Neutrosophic number. The score function and the removal area method are used to transform the Nonagonal Neutrosophic number to crisp number and the results obtained in both the methods are compared.

Dynamic Pricing and Inventory Control with Fixed Ordering Cost and Incomplete Demand Information

We consider the periodic review dynamic pricing and inventory control problem with fixed ordering cost. Demand is random and price dependent, and unsatisfied demand is backlogged. With complete demand information, the celebrated [Formula: see text] policy is proved to be optimal, where s and S are the reorder point and order-up-to level for ordering strategy, and [Formula: see text], a function of on-hand inventory level, characterizes the pricing strategy. In this paper, we consider incomplete demand information and develop online learning algorithms whose average profit approaches that of the optimal [Formula: see text] with a tight [Formula: see text] regret rate. A number of salient features differentiate our work from the existing online learning researches in the operations management (OM) literature. First, computing the optimal [Formula: see text] policy requires solving a dynamic programming (DP) over multiple periods involving unknown quantities, which is different from the majority of learning problems in OM that only require solving single-period optimization questions. It is hence challenging to establish stability results through DP recursions, which we accomplish by proving uniform convergence of the profit-to-go function. The necessity of analyzing action-dependent state transition over multiple periods resembles the reinforcement learning question, considerably more difficult than existing bandit learning algorithms. Second, the pricing function [Formula: see text] is of infinite dimension, and approaching it is much more challenging than approaching a finite number of parameters as seen in existing researches. The demand-price relationship is estimated based on upper confidence bound, but the confidence interval cannot be explicitly calculated due to the complexity of the DP recursion. Finally, because of the multiperiod nature of [Formula: see text] policies the actual distribution of the randomness in demand plays an important role in determining the optimal pricing strategy [Formula: see text], which is unknown to the learner a priori. In this paper, the demand randomness is approximated by an empirical distribution constructed using dependent samples, and a novel Wasserstein metric-based argument is employed to prove convergence of the empirical distribution. This paper was accepted by J. George Shanthikumar, big data analytics.

Efficient valuation of variable annuity portfolios with dynamic programming

AbstractThe valuation of variable annuity portfolios presents major challenges for US life insurers. Recent studies propose machine learning and metamodeling techniques based on selecting a few representative guarantees. However, these methods face a critical trade‐off between speed and accuracy. In contrast, I propose a recursive dynamic programming approach and demonstrate its ability to value a large and highly heterogeneous variable annuity portfolio with a high degree of accuracy and within a few seconds—even under stochastic interest rates and volatility—since the heavy computational burden can be fully front‐loaded (in a one‐time effort at the guarantee's pricing stage). This makes the dynamic programming approach ideally suited for all variable annuity applications, including the computation of reserves and capital requirements and to determine the insurer's hedging position. Moreover, dynamic programming can naturally incorporate optimal policyholder behavior into the insurer's valuation.

Dynamic Programming Analysis for Measuring the Optimization of the Manpower per Shift PT. XYZ

The problems that occur in the company is the production system make to order, for example PT. XYZ. Unknown certainty the amount of the order lenses will be produced. So it is not known how labor should be allocated to each shift. Then used the approach of dynamic programming to optimize the number of workers per shift in the PT. XYZ. This study aims to determine the analysis of the dynamic programming optimization to allocate the amount of labor per shift to be effective. The type of research data is primary data obtained through direct observation in PT. XYZ. The results of the research based on the calculation of the forward recursive dynamic programming: shift 1 = 105 people, shift 2 = 104 people and shift 3 = 104 people. While the solution of the optimal allocation of labor for each shift based on the calculation of the backward recursive dynamic programming: shift 1 = 105 people, shift 2 = 104 people and Shift 3 = 104 people. Then the PT. XYZ should allocate 313 workers, so that the allocation of labor to each shift is optimal according to the order received

Exact Methods for the Traveling Salesman Problem with Drone

Efficiently handling last-mile deliveries becomes more and more important nowadays. Using drones to support classical vehicles allows improving delivery schedules as long as efficient solution methods to plan last-mile deliveries with drones are available. We study exact solution approaches for some variants of the traveling salesman problem with drone (TSP-D) in which a truck and a drone are teamed up to serve a set of customers. This combination of truck and drone can exploit the benefits of both vehicle types: the truck has a large capacity but usually low travel speed in urban areas; the drone is faster and not restricted to street networks, but its range and carrying capacity are limited. We propose a compact mixed-integer linear program (MILP) for several TSP-D variants that is based on timely synchronizing truck and drone flows; such an MILP is easy to implement but nevertheless leads to competitive results compared with the state-of-the-art MILPs. Furthermore, we introduce dynamic programming recursions to model several TSP-D variants. We show how these dynamic programming recursions can be exploited in an exact branch-and-price approach based on a set partitioning formulation using ng-route relaxation and a three-level hierarchical branching. The proposed branch-and-price can solve instances with up to 39 customers to optimality outperforming the state-of-the-art by more than doubling the manageable instance size. Finally, we analyze different scenarios and show that even a single drone can significantly reduce a route’s completion time when the drone is sufficiently fast.

Efficient Valuation of Variable Annuity Portfolios with Dynamic Programming

The valuation of variable annuity (VA) portfolios presents major challenges for life insurers. Recent studies propose approximation methods based on selecting a few representative guarantees. In contrast, I present a bottom-up valuation approach using recursive dynamic programming (RDP). An in-depth numerical analysis shows that the RDP estimation is able to value a large VA portfolio with a high degree of accuracy and within a few seconds---even under stochastic interest rates and volatility---since the heavy computational burden can be fully front-loaded (in a one-time effort at the guarantee's pricing stage). The RDP approach outperforms competing methods in both speed and accuracy and is thus ideally suited for various VA-related applications, including the computation of GAAP reserves, statutory reserves and capital requirements, as well as to determine the insurer's hedging position. Moreover, RDP can naturally incorporate optimal policyholder behavior into the insurer's valuation.

Deep Neural Networks Algorithms for Stochastic Control Problems on Finite Horizon: Convergence Analysis

This paper develops algorithms for high-dimensional stochastic control problems based on deep learning and dynamic programming. Unlike classical approximate dynamic programming approaches, we first approximate the optimal policy by means of neural networks in the spirit of deep reinforcement learning, and then the value function by Monte Carlo regression. This is achieved in the dynamic programming recursion by performance or hybrid iteration, and regress now methods from numerical probabilities. We provide a theoretical justification of these algorithms. Consistency and rate of convergence for the control and value function estimates are analyzed and expressed in terms of the universal approximation error of the neural networks, and of the statistical error when estimating network function, leaving aside the optimization error. Numerical results on various applications are presented in a companion paper (arxiv.org/abs/1812.05916) and illustrate the performance of the proposed algorithms.

Online policy iteration ADP-based attitude-tracking control for hypersonic vehicles

An online adaptive dynamic programming (ADP) attitude-tracking controller based on policy iteration is proposed, aiming to approach the optimal control of hypersonic vehicles (HVs). The Bellman equation, known as the principal recursive dynamic programming formula, is provided to obtain the controller. In particular, the control action is generated by the ADP controller to track the attitude trajectory. In order to approach optimal control in the uncertain nonlinear HVs system, we use policy iteration to approximate the Bellman equation and build an actor-predictor-critic framework, in which the action network, state estimator and critic network are adopted to implement the policy iteration. Meanwhile, an offline learning method is provided to approach the initial value of iterative computations and improve the efficiency of online learning. The comparative simulations demonstrate the good performance of PIADP with aerodynamic parameter perturbations and random disturbances.

Dynamic Pricing and Inventory Control with Fixed Ordering Cost and Incomplete Demand Information

We consider the periodic review dynamic pricing and inventory control problem with fixed ordering cost. Demand is random and price dependent, and unsatisfied demand is backlogged. With complete demand information, the celebrated (s,S,p) policy is proved to be optimal, where s and S are the reorder point and order-up-to level for ordering strategy, and p, a function of on-hand inventory level, characterizes the pricing strategy. In this paper, we consider incomplete demand information and develop online learning algorithms whose average profit approaches that of the optimal (s,S,p) with a tight O (√T) regret rate. A number of salient features differentiate our work from the existing online learning researches in the OM literature. First, computing the optimal (s,S,p) policy requires solving a dynamic programming (DP) over multiple periods involving unknown quantities, which is different from the majority of learning problems in operations management that only require solving single-period optimization questions. It is hence challenging to establish stability results through DP recursions, which we accomplish by proving uniform convergence of the profit-to-go function. The necessity of analyzing action-dependent state transition over multiple periods resembles the reinforcement learning question, considerably more difficult than existing bandit learning algorithms. Second, the pricing function p is of infinite dimension, and approaching it is much more challenging than approaching a finite number of parameters as seen in existing researches. The demand-price relationship is estimated based on upper confidence bound, but the confidence interval cannot be explicitly calculated due to the complexity of the DP recursion. Finally, due to the multi-period nature of (s,S,p) policies the actual distribution of the randomness in demand plays an important role in determining the optimal pricing strategy p, which is unknown to the learner a priori. In this paper, the demand randomness is approximated by an empirical distribution constructed using dependent samples, and a novel Wasserstein metric based argument is employed to prove convergence of the empirical distribution.

Position Bidding for Algorithm-Assisted Drivers

Problem definition: We propose and analyze a novel bidding mechanism that, based on new 5G-based inter-vehicle communication technologies, brings economic efficiency to ad-hoc and unexpected traffic congestion points, while avoiding negative externalities to anyone involved. Academic/Practical Relevance: Road traffic delays and congestion is one of the most vexing problems of modern times. Many price-based approaches, such as tolls and zone-congestion pricing, have been implemented in practice and studied in the literature. However, such approaches are not applicable to ad-hoc and unexpected congestion points that frequently occur; concretely in this study, forced merging. We show how one can achieve economic efficiency in these situations, by combining the potential of new technologies with dynamic pricing. Methodology: We analyze the performance of a mechanism that bids sequentially from tail to head of a dense platoon of vehicles (T2H), with the condition that the vehicle can advance only if it wins the bid, imposing, thus, no negative externalities. For benchmarking, we also analyze a mechanism that bids sequentially from head to tail (H2T), clearly favoring the merging driver. In both cases, we show that optimal bids and value functions can be computed via simple Dynamic Programming recursions. Results: For uniformly distributed time valuations, we prove that the expected social welfare of T2H is close to a partial-information social optimum, and that the expected social welfare of H2T is lower than that of T2H, as long as the platoon is not too short. We also propose fast Approximate Dynamic Programming algorithms that perform very well in our numerical experiments, making them suitable for implementation. Managerial Implications: This is a novel proposal for the future management of algorithm-assisted, connected road traffic, which has the potential to achieve economic efficiency without negative externalities to anyone involved; the latter being critical for mass adoption.

Optimal Resource Allocation for Multicarrier NOMA in Short Packet Communications

In this paper, we study the resource allocation problem for downlink multicarrier non-orthogonal multiple access systems with short-packet communications (MC-NOMA-SPC). In contrast to long-packet communications in conventional wireless systems, SPC suffers from a transmission rate degradation and an unavoidable decoding error rate. Thus conventional resource allocation based on the Shannon capacity assuming infinite blocklength is no longer optimal. In this paper, we propose to use the effective-throughput as the performance metric to evaluate the tradeoff between the transmission rate and the decoding error rate. Building on that, we jointly optimize the subcarrier assignment, transmission power allocation, and transmission rate adaptation of each user to maximize the total weighted effective-throughput subject to transmission reliability and various practical constraints. Since the formulated problem belongs to a non-convex mixed integer nonlinear programming (MINLP) problem, we develop an efficient algorithm based on the dynamic programming (DP) recursion framework to obtain its optimal solutions. We further propose a low-complexity algorithm based on the principles of block coordinate descent (BCD) and concave-convex procedure (CCCP). Finally, the simulation results show that the proposed low-complexity algorithm can achieve similar performance as that of the optimal solution and outperform the other baseline schemes significantly.

Optimal Design of Gravity-Fed Sewer Lines Using Linear Programming

The cost of a sewerage system is mainly governed by the size of the sewer pipe, excavation depth and manhole spacing. A linear programming model is developed to minimize the total cost comprising of the pipeline cost, excavation cost and manhole cost of the sewer line. The constraints of the optimization model are related to the distance between two consecutive manholes, and slope of the sewer line to maintain the self-cleansing velocity. The nonlinearity due to the pipe size is eliminated by considering only those available diameters that satisfy the self-cleansing velocity constraint. The model selects the combination of pipe sizes and slope of the sewer line between different manholes maintaining the self-cleansing velocity, which results in the minimum value of the total cost of the entire sewer line. The application of the developed model is illustrated with the help of an existing design problem, and the results are compared with the available solution using forward recursive dynamic programming. It is found that the linear programming model results in lesser value of the total cost of the sewer line.