Two-stage Dynamic Programming Research Articles

Improving new energy subsidy efficiency considering learning effect: A case study on wind power

Subsidy policies are instrumental in driving the development of new energy. However, the effective allocation of new energy subsidies over time is challenging given fiscal constraints. This study addresses this challenge by considering the learning effect associated with the new energy industry. A two-stage dynamic programming model is proposed to capture the investment decision-making process of companies under new energy subsidy policies and government subsidy setups. Theoretical findings suggest that company investment decisions in new energy are influenced by a guiding principle: The subsidy rate should be negatively correlated with the variation rate of production scale increment (VRPSI). We calibrate this investment decision principle using wind power data from 14 countries. According to this principle, excessive subsidy rates may result in a low VRPSI, thereby diminishing future investment profitability in the new energy industry and leading to subsidy inefficiency. Upon investigating the efficiency of annual subsidy allocation, we find that the subsidy rates were potentially set too high in 2014, 2016, and 2017. Furthermore, the government should exercise caution regarding an inefficient subsidy pattern whereby companies invest in new energy only when the subsidy rate exceeds a certain threshold, neglecting traditional power sources. It is crucial to note that although this study uses wind power industry data for calibration and simulation, the theoretical model can be broadly applied to other new energy industries and emerging industries with increasing marginal net profit.

Data-Based Optimal Switching and Control With Admissibility Guaranteed Q -Learning.

This article addresses the data-based optimal switching and control codesign for discrete-time nonlinear switched systems via a two-stage approximate dynamic programming (ADP) algorithm. Through offline policy improvement and policy evaluation, the proposed algorithm iteratively determines the optimal hybrid control policy using system input/output data. Moreover, a strict proof of the convergence is given for the two-stage ADP algorithm. Admissibility, an essential property of the hybrid control policy must be ensured for practical application. To this end, the properties of the hybrid control policies are analyzed and an admissibility criterion is obtained. To realize the proposed Q -learning algorithm, an actor-critic neural network (NN) structure that employs multiple NNs to approximate the Q -functions and control policies for different subsystems is adopted. By applying the proposed admissibility criterion, the obtained hybrid control policy is guaranteed to be admissible. Finally, two numerical simulations verify the effectiveness of the proposed algorithm.

Two-Stage Dynamic Programming in the Routing Problem with Decomposition

Two-Stage Dynamic Programming in the Routing Problem with Decomposition

Distribution Network Repair Scheduling With Multiple Types of Cooperation Using Dynamic Programming after Disasters

Distribution of network repair scheduling after disasters (DNRSD) is important for developing a reasonable repair strategy and improving the efficiency of distribution of network repair after disasters. This study considers two cooperative relationships in the repair model, namely, simultaneous cooperation (SC) and ordered cooperation (OC), and proposes a new post-disaster distribution network repair model, the multiple cooperation model (MCM). Compared with conventional modeling, MCM can help to adjust repair strategies and prevent repair strategies from being delayed while in implementation. The original problem is divided into two stages, and collaboratively solved via dynamic programming. Compared with conventional solving methods, the proposed two-stage dynamic programming (TDP) optimization method can obtain global optimal solutions and make more reasonable repair strategies. Test cases based on the modified IEEE 123-bus and 8500-bus distribution systems verify the advantages of our proposed modeling and optimization method.

Metamodeling the optimal total revenues of the short-term optimization of a hydropower cascade under uncertainty

This paper deals with the optimization of the short-term production planning of a real cascade of run-of-river hydropower plants. Water inflows and electricity prices are subject to data uncertainty and they are modeled by a finite set of joint scenarios. The optimization problem is written with a two-stage stochastic dynamic mixed-integer linear programming formulation. This problem is solved by replacing the value function of the second stage with a surrogate model. We propose to evaluate the feasibility of fitting the surrogate model by supervised learning during a pre-processing step. The learning data set is constructed by Latin hypercube sampling after discretizing the functional inputs. The surrogate model is chosen among linear models and the dimension of the functional inputs is reduced by principal components analysis. Validation results for one simplified case study are encouraging. The methodology could however be improved to reduce the prediction errors and to be compatible with the time limit of the operational process.

To the application of two-stage dynamic programming in the problem of sequential visiting of megalopolises

It is considered the routing problem solution in which the task set is the union of two subsets (zones); one of subsets should be serviced earlier than the second. It is supposed that each task is connected with the visit to a megalopolis (nonempty finite set) and fulfilment of works. The regular succession of task fulfilment should satisfy to precedence conditions realized in each of zones. The cost functions assume a task list dependence; cost aggregation is supposed additive. For solution, the two-stage dynamic programming procedure is used.

Optimal operation framework of customer-premise battery storage for energy charge reduction and primary frequency regulation

This paper focuses on the intra-day operation of customer-premise battery storage for multiple applications, in particular energy charge reduction and primary frequency regulation (PFR). The nature of this problem requires the battery to provide reliable PFR, as well as coordinate online optimization performance with computing time while accounting for multidimensional stochastic information. Therefore, we propose a novel optimal operation framework. In the day-ahead phase, the piecewise linear functions (PLFs) are trained offline to approximate the expected operating cost of the long-term future. During the intra-day operation, a two-stage approximate dynamic programming (ADP) model is formulated utilizing the updated short-term future information and the PLFs based long-term estimation. To ensure the reliability of frequency regulation, we develop a robust correction strategy of PFR signals. Combining the correction feedback, the two-stage ADP model is optimized through rolling horizon procedures to dynamically adjust the power basepoint. Numerical simulations demonstrate the performance of our proposed operation framework. The convergence rate of PLFs offline training is 97.3%. In addition, the intra-day operating cost of our proposed framework is merely 1.62% higher than the theoretical optimum, while the online calculation takes only 3.25 s for one rolling period.

Two-Stage Dual Dynamic Programming With Application to Nonlinear Hydro Scheduling

We present an approximate method for solving nonlinear control problems over long time horizons, in which the full nonlinear model is preserved over an initial part of the horizon, while the remainder of the horizon is modeled using a linear relaxation. As this approximate problem may still be too large to solve directly, we present a Benders decomposition-based solution algorithm that iterates between solving the nonlinear and linear parts of the horizon. This extends the dual dynamic programming approach commonly employed for optimization of linearized hydro power systems. We prove that the proposed algorithm converges after a finite number of iterations, even when the nonlinear initial stage problems are solved inexactly. We also bound the suboptimality of the split-horizon method with respect to the original nonlinear problem, in terms of the properties of a map between the linear and nonlinear state-input trajectories. We then apply this method to a case study concerning a multiple reservoir hydro system, approximating the nonlinear head effects in the second stage using McCormick envelopes. We demonstrate that near-optimal solutions can be obtained in a shrinking horizon setting when the full nonlinear model is used for only a short initial section of the horizon. For this example, the approach is shown to be more practical than both conventional dynamic programming and a multi-cell McCormick envelope approximation from the literature.

An Improved Lagrangian Relaxation Algorithm for the Robust Generation Self-Scheduling Problem

The robust generation self-scheduling problem under electricity price uncertainty is usually solved by the commercial solver, which is limited in computation time and memory requirement. This paper proposes an improved Lagrangian relaxation algorithm for the robust generation self-scheduling problem where the quadratic fuel cost and the time-dependent exponential startup cost are considered. By using the optimal duality theory, the robust generation self-scheduling problem, which has a max-min structure, is reformulated as a minimization mixed integer nonlinear programming (MINLP) problem. Upon the reformulation, the Lagrangian relaxation algorithm is developed. To obtain a solvable relaxed problem, the variable splitting technique is introduced before the relaxation. The obtained relaxed problem is decomposed into a linear programming-type subproblem and multiple single-unit subproblems. Each single-unit subproblem is solved optimally by a two-stage backward dynamic programming procedure. The special cases of the problem are discussed and a two-stage algorithm is proposed. The proposed algorithms are tested on test cases of different sizes and the numerical results show that the algorithms can find near-optimal solutions in a reasonable time.

Optimal Management for Parking-Lot Electric Vehicle Charging by Two-Stage Approximate Dynamic Programming

This paper targets the day-time charging scenario for plug-in electric vehicles at parking-lots near commercial places, where most vehicles have extended parking time. Compared with night-time charge scenarios for residential buildings, commercial building parking-lot charging during day-time feature significant stochastic vehicle arrival and departure, as well as highly dynamic electricity price. A two-stage approximate dynamic programming framework is proposed to determine the optimal charging strategy, utilizing the predicted short-term future information and long-term estimation from historical data. All the vehicles are desired to be charged to full prior to the departure time specified under constrained total charging capacity. The uncharged amount is subject to a significant penalty cost. Simulation scenarios are created by modeling the vehicle arrival behavior as Poisson process, including arrival time, departure time, and arrival state of charge. The simulation results show that the proposed method can significantly decrease the energy cost.

Aggregate\u2013disaggregate approach to an airfreight forwarder\u2019s planning under uncertainty: a case study

Resource planning of airfreight forwarders is a complex endeavor, requiring decisions to be made in a dynamic and uncertain environment. Airfreight forwarders acquire airfreight spaces from three sources: allotment from carriers, retail from carriers and subcontracting to partners, all of which differ in terms of cost, flexibility and timing of booking. This real-life problem has many planning decisions (bookings in terms of carriers, route, time, ULDs, etc.). In this case study, we propose an aggregate–disaggregate approach and focus on the most critical decisions. A two-stage stochastic dynamic program first determines, in aggregates, the amount of allotment bookings, retail resources, and subcontracting or surplus co-loading. Then, a heuristic is used to disaggregate resource requirements into specific bookings. An analysis is provided to examine the relevant managerial issues. Based on real-life data, we show several patterns of aggregate resource bookings with respect to different levels of demand uncertainty and cost parameters. We show that resource disaggregation has to balance cost-effectiveness, capacity flexibility and routing flexibility of a resource plan.

Scheduling for yard cranes based on two-stage hybrid dynamic programming

Making operational plans for Yard Cranes (YCs) to enhance port efficiency has become vital issues for the container terminals. This paper discusses the load-scheduling problem of multiple YCs. The problem is to schedule two YCs at different container blocks, which serve the loading operations of one quay crane so as to minimize the total distance of visiting paths and the make-span at stack area. We consider the container handling time, the YC visiting time, and the waiting time of each YC when evaluating the make-span of the loading operation by YCs. Both the container bay visiting sequences and the number of containers picked up at each visit of the two YCs are determined simultaneously. A mathematical model, which considers interference between adjacent YCs, is provided by means of time-space network to formulate the problem and a two-stage hybrid algorithm composed of greedy algorithm and dynamic programming is developed to solve the proposed model. Numerical experiments were conducted to compare performances of the algorithm in this study with actual scheduling rules.

Balancing pre-disaster preparedness and post-disaster relief

Challenges associated with resource allocation to mitigate and recover from natural and man-made disasters inspire new theoretical questions for decision making in the intertwined natural and human world. Disaster loss is determined not only by post-disaster relief but also the pre-disaster mitigation and preparedness. To examine the decision making process at ex ante and ex post disaster stages, we develop a two-stage dynamic programming model that optimally allocates preparedness and relief expenditures. We analytically and numerically solve the model and provide new insights by sensitivity analysis.

Integrated Staff and Room Scheduling for Surgeries: Methodology and Application

We consider the problem of minimizing daily expected resource usage and overtime costs across multiple parallel resources such as anesthesiologists and operating rooms, which are used to conduct a variety of surgical procedures at large multi specialty hospitals. To address this problem, we develop a two-stage stochastic dynamic programming model with recourse. The first stage allocates these resources across multiple surgeries with uncertain durations and prescribes the sequence of surgeries to these resources. The second stage determines actual start times to surgeries based on realized durations of preceding surgeries and assigns overtime to resources to ensure all surgeries are completed using the allocation and sequence determined in the first stage. We develop a data driven robust optimization method that solves large-scale real-sized versions of this model close to optimality. We validate and implement this model as a decision support system at the UCLA Ronald Reagan Medical Center. This has led to an average daily cost savings of around 7% or $2.2 million on an annual basis. In addition, the insights based on this model have significantly influenced decision making at the operating services department at this hospital.

Dual sourcing and backup production: Coexistence versus exclusivity

Supply risk is one of the most important enterprise risks. Dual sourcing and backup production are the two operational strategies that firms most commonly use to mitigate supply risk. The base model examines the strategic use of dual sourcing and backup supply options in a two-stage dynamic programming model for one period. In the first stage the firm decides whether to employ single or dual sourcing for primary supply and whether to reserve backup option. In the second stage, if backup option has been reserved, the firm decides the backup production quantity after observing the realized primary delivery. The optimal sourcing strategies are separately characterized under deterministic and uncertain demand. The results show that dual sourcing coexists with backup production when the two primary suppliers have similar and moderate reliabilities and the backup reservation cost is relatively small. When the reliabilities of the two primary suppliers differ significantly, dual sourcing and backup can become mutually exclusive strategies. Furthermore, the existence of the backup production option decreases the probability of selecting dual sourcing; compared with deterministic demand, uncertain demand increases the area of coexistence of dual sourcing and backup production. In particular, when the firm employs dual sourcing, backup production is activated under deterministic demand only when both primary suppliers fail, while can be activated under uncertain demand when one of the primary suppliers fails. We extend the base model to a two-period structure where the reliability of one primary supplier is uncertain, and the firm updates its assessment of supplier reliability based on the delivery in previous period. Based on Bayesian forecast-updating approach, we find that reliability learning weakly increases the area of coexistence of dual sourcing and backup, and weakly decreases the probability of contingent backup production in the coexistence area.

Base Station Sleeping and Resource Allocation in Renewable Energy Powered Cellular Networks

We consider energy-efficient wireless resource management in cellular networks where BSs are equipped with energy harvesting devices, using statistical information for traffic intensity and harvested energy. The problem is formulated as adapting BSs' on-off states, active resource blocks (e.g. subcarriers) as well as power allocation to minimize the average grid power consumption in a given time period while satisfying the users' quality of service (blocking probability) requirements. It is transformed into an unconstrained optimization problem to minimize a weighted sum of grid power consumption and blocking probability. A two-stage dynamic programming (DP) algorithm is then proposed to solve this optimization problem, by which the BSs' on-off states are optimized in the first stage, and the active BS's resource blocks are allocated iteratively in the second stage. Compared with the optimal joint BSs' on-off states and active resource blocks allocation algorithm, the proposed algorithm greatly reduces the computational complexity, while at the same time achieves close to the optimal energy saving performance.

A dynamic programming based heuristic for locating stops in public transportation networks

The length of journey towards a bus stop or railway station greatly influences passenger satisfaction and, thus, the utilization of public transport offers. In this context, we investigate the maximum covering location problem in networks (MCLPN) where stops (or stations) are to be located in a given railway or bus network, such that the number of passengers reaching a stop within their particular coverage radius is maximized. Up to now, no specialized solution procedure directly addressing MCLPN exists, instead it is mainly referred to the well-known maximum covering location problem (MCLP), which, however, neglects the underlaying information of the given network structure. This paper exploits the network information, introduces a fast and efficient two-stage dynamic programming based heuristic specifically tailored to MCLPN, and compares it to existing procedures for MCLP. The results show that our heuristic delivers near optimal solutions, i.e., 87% of our test instances are solved to optimality, within a few seconds of computational time.

An inexact two-stage dynamic stochastic model for regional electricity and heat supply management with pollutants mitigation control

Energy system management is an important tool for regional energy and environmental development, and many parameters and their interrelationships in energy-environmental management model appear complexity and uncertain. How to deal with these uncertainties and make a reasonable decision schemes are desired for managers. In this study, an inexact two-stage dynamic programming model is developed for regional electricity and heat supply management under considering the complexities and uncertainties in regional energy system. The model can reflect not only uncertainties expressed as probability distribution but also those being available as intervals. The developed model is applied to a case of planning regional electricity and heat supply as well as pollution emission reduction considered. A number of scenarios corresponding to different pollutants emission reduction levels are examined; the results indicated that reasonable solutions have been generated under different pollutants reduction levels. They can be used for generating plans for energy resource/electricity/heat allocation and capacity expansion and help decision makers identify desired regional electricity and heat supply which need minimum cost under various standards of pollutants emission reduction control.

An effective Lagrangian relaxation approach for rescheduling a steelmaking-continuous casting process

This paper studies a steelmaking-continuous casting (SCC) rescheduling problem with machine breakdown and processing time variations. Two objectives are considered in this study: the efficiency objective and the stability objective. The former refers to the total weighted completion time and total sojourn time, whereas the latter refers to the number of operations processed on different machines in the initial and revised schedules. We develop a time-index formulation and an effective Lagrangian relaxation (LR) approach with machine capacity relaxation to address the rescheduling problem. The LR approach decomposes the relaxed problem into batch-level subproblems with variable processing times. A polynomial two-stage dynamic programming algorithm is proposed to solve the batch-level subproblems. An efficient subgradient algorithm with global convergence is presented to solve the corresponding Lagrangian dual (LD) problem. Computational experiments based on practical production data show that the proposed approach not only produces a high quality schedule within an acceptable time but also performs much better than a practical SCC rescheduling method from a large iron and steel enterprise in China.

The Study on the Value of Information Sharing with Correlated Market Demand and Cost of Information Sharing in Supply Chain

In a two-echelon supply chain system with 1 manufacture and N retailers, the two-stage stochastic dynamic program of whether manufacture to share information with n (0≤n ≤ N) retailers is constructed when the market demand is correlated with AR (1) module and information sharing is costly. The value of information shared with n retailers is analyzed analytically. One numerical case is simulated to illustrate the magnitude of the value of information shared with n retailers in the supply chain at last.