Dynamic Programming Methodology Research Articles

Approximate dynamic programming for condition-based node deployment in a wireless sensor network

The flexibility of deployment strategies combined with the low cost of individual sensor nodes allow wireless sensor networks (WSNs) to be integrated into a variety of applications. Network operations degrade over time as sensors consume a finite power supply and begin to fail. In this work we address the selective maintenance of a WSN through a condition-based deployment policy (CBDP) in which sensors are deployed over a series of missions. The main contribution is a Markov decision process (MDP) model to maintain a reliable WSN with respect to region coverage. Due to the resulting high dimensional state and outcome space, we explore approximate dynamic programming (ADP) methodology in the search for high quality CBDPs. Our model is one of the first related to the selective maintenance of a large-scale WSN through the repeated deployment of new sensor nodes with a reliability objective, and one of the first ADP applications for the maintenance of a complex WSN. Additionally, our methodology incorporates a destruction spectrum reliability estimate which has received significant attention with respect to network reliability, but its value in a maintenance setting has not been widely explored. We conclude with a discussion on CBDPs in a range of test instances, and compare the performance to alternative deployment strategies.

Dynamic Programming Approaches for Resource Allocation in Project Scheduling: Maximizing Efficiency under Time and Budget Constraints

This study investigates dynamic programming methodologies for resource allocation in project scheduling, aiming to optimize efficiency while adhering to time and budget limitations. Exploring the theoretical underpinnings of dynamic programming, including time and budget restrictions into resource allocation models, and conducting case studies to assess actual applications are the primary goals of this project. This study utilized a thorough review process, including a synthesis of the relevant literature, an analysis of the case study, and policy implications. Among the most important discoveries are the following: the versatility of dynamic programming techniques in addressing resource allocation challenges across industries, the significance of incorporating time and budget constraints into decision-making processes, and the necessity of addressing limitations related to computational complexity, data requirements, and risk management. In the context of policy consequences, investments in computing infrastructure, data management techniques, and risk mitigation strategies are highlighted. In general, the findings of this study highlight the potential of dynamic programming methodologies to improve the efficiency and effectiveness of resource allocation within the context of a project that is affected by time and financial restrictions.

Battery Dispatching for End Users With On-Site Renewables and Peak Demand Charges—An Approximate Dynamic Programming Approach

Battery energy storage systems (BESSs) have the potential to reduce end users’ electricity bills by shifting their grid demand in response to price incentives. Some electricity retailers charge end users for their peak demand in the billing cycle regardless of the time of occurrence. Yet, using BESSs to reduce demand charges is challenging, since the net demand of end users can be highly uncertain due to low aggregation and on-site renewable generation. In this article, an approximate dynamic programming (ADP) methodology is developed to control a BESS to minimize an end user’s electricity bill, which includes both an energy charge and monthly peak demand charge. To address time-varying uncertainty, the net demand is modeled using a periodic autoregressive (PAR) model, which is then used to formulate a Markov Decision Process whose objective is to minimize the sum of energy, peak demand, and battery usage costs. A backward ADP strategy is developed which is enabled by new closed-form expressions for the probability distribution of the expected stage and tail costs when a radial basis function (RBF) value function approximation (VFA) is employed. The approach is applied to real net demand data from a small Australian residential community and compared to two benchmark policies: a lookup table VFA and a model predictive control (MPC) approach. The results show that the proposed approach reduces the average monthly peak demand by about 25% (yielding an 8% reduction in the electricity bill), whereas the best-performing benchmark policy reduced the peak demand by about 17%.

An approximate dynamic programming algorithm for short-term electric vehicle fleet operation under uncertainty

This paper considers the dynamic problem of optimally operating a fleet of plug-in hybrid electric vehicles in a market environment. With uncertainty in future electricity prices and driving demands, we formulate a Markov decision process and determine a cost-minimizing policy for using the engine and charging and discharging the battery. As such, the policy is based on the trade-off between the costs of gasoline and electricity and between current and future power prices. To accommodate an inhomogeneous fleet composition and overcome the computational challenges of stochastic and dynamic optimization, including large-scale state and action spaces, we adopt the methodology of approximate dynamic programming. More specifically, using simulation and value function approximation by linear regression, we apply a least squares Monte Carlo method. This methodology allows for scaling with respect to fleet size and we are able to establish convergence of our algorithm for 100 vehicles by using 5000 samples in the simulation. Our results show that the vehicles should generally discharge the battery rather than using the engine unless battery capacity is insufficient to fully cover driving demand, but the timing of battery charging should be according to power prices. When comparing our policy to the simple policy of immediate charging, we demonstrate superiority for small and medium-sized fleets, with 2%–4% cost differences.

Dynamic programming approach for fuzzy linear programming problems FLPs and its application to optimal resource allocation problems in education system

This research is about the development of a dynamic programming model for solving fuzzy linear programming problems. Initially, fuzzy dynamic linear programming model FDLP is developed. This research revises the established dynamic programming model for solving linear programming problems in a crisp environment. The mentioned approach is upgraded to address the problem in an uncertain environment. Dynamic programming model can either be passing forward or backward. In the proposed approach backward dynamic programming approach is adopted to address the problem. It is then followed by implementing the proposed method on the education system of Pakistan. The education system of Pakistan comprises of the Primary, Middle, Secondary, and Tertiary education stages. The problem is to maximize the efficiency of the education system while achieving the targets with minimum usage of the constrained resources. Likewise the model tries to maximize the enrollment in the Primary, Middle, Secondary and Tertiary educational categories, subject to the total available resources in a fuzzy uncertain environment. The solution proposes that the enrollment can be increased by an amount 9997130, by increasing the enrollment in the Middle and Tertiary educational categories. Thus the proposed method contributes to increase the objective function value by 30%. Moreover, the proposed solutions violate none of the constraints. In other words, the problem of resources allocation in education system is efficiently managed to increase efficiency while remaining in the available constrained resources. The motivation behind using the dynamic programming methodology is that it always possesses a numerical solution, unlike the other approaches having no solution at certain times. The proposed fuzzy model takes into account uncertainty in the linear programming modeling process and is more robust, flexible and practicable.

Lookahead approximate dynamic programming for stochastic aircraft maintenance check scheduling optimization

This paper proposes a lookahead approximate dynamic programming methodology for aircraft maintenance check scheduling, considering the uncertainty of aircraft daily utilization and maintenance check elapsed time. It adopts a dynamic programming framework, using a hybrid lookahead scheduling policy. The hybrid lookahead scheduling policy makes the one-step optimal decision for heavy aircraft maintenance based on deterministic forecasts and then determines the light maintenance according to stochastic forecasts. The objective is to minimize the total wasted utilization interval between maintenance checks while reducing the need for additional maintenance slots. By achieving this goal, one is also reducing the number of maintenance checks and increasing aircraft availability while respecting airworthiness regulations. We validate the proposed methodology using the fleet maintenance data from a major European airline. The descriptive statistics of several test runs show that, when compared with the current practice, the proposed methodology potentially reduces the number of A-checks by 1.9%, the number of C-checks by 9.8%, and the number of additional slots by 78.3% over four years.

Predicting morbidity by local similarities in multi-scale patient trajectories

Patient Trajectories (PTs) are a method of representing the temporal evolution of patients. They can include information from different sources and be used in socio-medical or clinical domains. PTs have generally been used to generate and study the most common trajectories in, for instance, the development of a disease. On the other hand, healthcare predictive models generally rely on static snapshots of patient information. Only a few works about prediction in healthcare have been found that use PTs, and therefore benefit from their temporal dimension. All of them, however, have used PTs created from single-source information. Therefore, the use of longitudinal multi-scale data to build PTs and use them to obtain predictions about health conditions is yet to be explored. Our hypothesis is that local similarities on small chunks of PTs can identify similar patients concerning their future morbidities. The objectives of this work are (1) to develop a methodology to identify local similarities between PTs before the occurrence of morbidities to predict these on new query individuals; and (2) to validate this methodology on risk prediction of cardiovascular diseases (CVD) occurrence in patients with diabetes. We have proposed a novel formal definition of PTs based on sequences of longitudinal multi-scale data. Moreover, a dynamic programming methodology to identify local alignments on PTs for predicting future morbidities is proposed. Both the proposed methodology for PT definition and the alignment algorithm are generic to be applied on any clinical domain. We validated this solution for predicting CVD in patients with diabetes and we achieved a precision of 0.33, a recall of 0.72 and a specificity of 0.38. Therefore, the proposed solution in the diabetes use case can result of utmost utility to secondary screening.

Dynamic optimization for multi-goals wealth management

We develop a dynamic programming methodology that seeks to maximize investor outcomes over multiple, potentially competing goals (such as upgrading a home, paying college tuition, or maintaining an income stream in retirement), even when financial resources are limited. Unlike Monte Carlo approaches currently in wide use in the wealth management industry, our approach uses investor preferences to dynamically make the optimal determination for fulfilling or not fulfilling each goal and for selecting the investor’s investment portfolio. This can be computed quickly, even for numerous investor goals spread over different or concurrent time periods, where each goal may be all-or-nothing or may allow for partial fulfillment. The probabilities of attaining each (full or partial) goal under the optimal scenario are also computed, so the investor can ensure the algorithm accurately reflects their preference for the relative importance of each of their goals. This approach vastly outperforms static portfolio strategies and target-date funds, widely used in the wealth management industry.

Optimization of energy and economic scheduling of a hybrid energy plant by using a dynamic programming approach

An optimized energy and economic scheduling of hybrid energy plants can lead to a significant reduction of primary energy consumption and operational costs. Various optimization methods, with their own advantages and limitations, have been proposed in the literature. However, the scheduling optimization of complex hybrid energy plants that are composed of renewable, conventional and storage energy technologies is still an area which demands contribution. Dynamic programming has proved to be a powerful approach because of its ability to solve a variety of optimization problems with nonlinear objective functions and constraints, as well as to find global optimal solutions. Thus, this paper goes beyond previous analysis available in the literature by developing a novel methodology based on dynamic programming for the optimization of the energy and economic scheduling of hybrid energy plants. The hybrid energy plant considered in this paper includes renewable energy systems, fossil fuel energy systems and energy storage technologies. The actual fluctuation of the electricity prices is also considered in this work. The optimal scheduling was identified by considering the minimization of primary energy consumption or operational costs, as well as a hybrid scenario for meeting thermal, cooling and electrical energy demands of the user. Hybrid scenarios of minimizing both primary energy consumption and operational costs weighted by two different weight coefficients α and β, are also evaluated. The validity and capability of the optimization methodology is demonstrated by considering two case studies. The first case is a commercial building and the second case regards a University campus. Compared to commonly-used operation strategies, the energy scheduling optimization (α = 1 and β = 0) by means of dynamic programming allows a primary energy saving between 3.8% and 8.3% for the first case study and a saving between 0.5% and 17.4% for the second case study. Moreover, the economic scheduling optimization (α = 0 and β = 1) enables operational cost reduction in the range 11.7%–25.1% for the first case study and in the range 4.3%–14% for the second case study. For both case studies, the economic scheduling optimization shows that fulfilling the user energy demands by a combined heat and power is economically more convenient than importing electricity from the grid. Finally, unlike the operation strategies used as benchmarks, the dynamic programming methodology is flexible and able to solve scheduling optimization problems under different optimization constraints and can also allow customized hybrid solutions.

Optimal decision-making for tall buildings in the aftershock environment

Aftershocks can exacerbate the financial losses and functionality disruption to buildings damaged by a preceding mainshock event. A framework is presented for determining the optimal set and timing of activities that are performed on an earthquake-damaged tall building in the aftershock environment. The optimization objective is to minimize the cumulative aftershock loss while considering the necessary safety precautions. To this end, the Markov Process and Dynamic Programming methodology are implemented. The specific post-mainshock actions that are considered include whether to evacuate the building, when to reoccupy the building if it was previously evacuated, whether to repair the building and if so, when to start and stop the repairs. The results for a 42-story reinforced concrete building show that the algorithm is able to balance the need for evacuation during the period of high aftershock seismic hazard with a timely resumption of occupancy and necessary repair activities to minimize functional disruption.

Control of blowup and approximate optimality conditions for the wave equations

We study the blowup problems for semilinear wave equations with control function in the domain and on the boundary. Sufficient optimality conditions for controlled blowup time are derived in terms of new dynamic programming methodology. We define ε-optimal value function and we construct sufficient ε-optimal conditions for that function again in terms of new dynamic programming inequality.

Sufficient optimality conditions for problems controlled by ordinary differential equations with behavior of blowup

SummaryWe study the blowup problem for nonautonomous systems of ordinary differential equations. Sufficient optimality conditions for the controlled blowup time are derived, in terms of a dual dynamic programming methodology. We define an ε‐optimal value function and we construct sufficient ε‐optimality conditions for that function again in terms of a dual dynamic programming inequality.

Stackelberg Differential Game-based Optimal Fault Estimation and Accommodation for Continuous-time Linear Systems

This paper deals with the optimal fault estimation and accommodation problem for a class of linear systems in the framework of Stackelberg differential game theory. In this framework, the observer plays the role of the follower, while the system plays the role of the leader in making sequential decisions. A dual controller approach is used to design an auxiliary controller for the observer such that it can non-cooperate with the controller of the system to achieve the Stackelberg equilibrium. To achieve the online updating of the fault-tolerant controller, an adaptive dynamic programming methodology is used by establishing two critic neural networks for the observer and system respectively. Finally, a simulation is presented to illustrate the efficiency and applicability of the theoretical results.

Dynamic Optimization for Goals-Based Wealth Management with Multiple Goals

We develop a dynamic programming methodology that seeks to maximize investor outcomes over multiple, potentially competing goals (such as upgrading a home, paying college tuition, or maintaining an income stream in retirement), even when financial resources are limited. Unlike Monte Carlo approaches currently in wide use in the wealth management industry, our approach uses investor preferences to dynamically make the optimal determination for fulfilling or not fulfilling each goal and for selecting the investor’s investment portfolio. This can be computed quickly, even for numerous investor goals spread over different or concurrent time periods, where each goal may allow for partial fulfillment or be all-or-nothing. The probabilities of attaining each (full or partial) goal under the optimal scenario are also computed, so the investor can ensure the algorithm accurately reflects their preference for the relative importance of each of their goals. These portfolio prescriptions are consistent with Prospect Theory.

Optimal blowup time of diffusion equations with control

ABSTRACT We study the blowup time for semilinear parabolic differential equations with control function by constructing new dynamic programming methodology. This new approach allows us to formulate verification theorem and sufficient optimality conditions of the first order for the controlled blowup time. We define an ϵ-optimal value function and we construct sufficient ϵ-optimal conditions for that function again in terms of new dynamic programming inequality.

Representation of Wind Energy Scenarios in the Mid-Term Hydrothermal Systems Operation Scheduling

This paper presents a modified stochastic dual dynamic programming methodology in order to include the wind energy scenarios in the mid-term hydrothermal systems operation planning with an hourly representation of the demand. As the wind power intermittence adds more complexity for this class of problem, the conventional stochastic dual dynamic programming (SDDP) is not suitable to solve it due the excessive computational effort. Then the proposed approach includes in a conventional SDDP an analytical representation of immediate cost function for each stage and inflow/wind scenarios. Analysis results considering different cases of the South Brazilian system will be used to show the effectiveness and robustness of the proposed methodology.

Sum Throughput Maximization in a Cognitive Multiple Access Channel With Cooperative Spectrum Sensing and Energy Harvesting

This paper focuses on the problem of sensing throughput optimization in a fading multiple access cognitive radio (CR) network, where the secondary user (SU) transmitters participate in cooperative spectrum sensing and are capable of harvesting energy and sharing energy with each other. We formulate the optimization problem as a maximization of the expected achievable sum-rate over a finite horizon, subject to an average interference constraint at the primary receiver, peak power constraints, and energy causality constraints at the SU transmitters. The optimization problem is a non-convex, mixed integer non-linear program (MINLP) involving the binary action to sense the spectrum or not, and the continuous variables, such as the transmission power, shared energy, and sensing time. The problem is analyzed under two different assumptions on the available information pattern: 1) non-causal channel state information (CSI), energy state information (ESI), and infinite battery capacity and 2) the more realistic scenario of the causal CSI/ESI and finite battery. In the non-casual case, this problem can be solved by an exhaustive search over the decision variable or an MINLP solver for smaller problem dimensions, and a novel heuristic policy for larger problems, combined with an iterative alternative optimization method for the continuous variables. The causal case with finite battery is optimally solved using a dynamic programming (DP) methodology, whereas a number of sub-optimal algorithms are proposed to reduce the computational complexity of DP. Extensive numerical simulations are carried out to illustrate the performance of the proposed algorithms. One of the main findings indicates that the energy sharing is more beneficial when there is a significant asymmetry between average harvested energy levels/channel gains of different SUs.

Method ofdynamic programming for calculation of managerial alternatives for stimulation of agricultural production

A method for substantiating managerial alternatives in stimulating agricultural production, based on the methodology of dynamic programming, was developed and tested. Approbation of the method included 8 consecutive stages. At the first stage, the grading of the studied indicators was carried out by the method of statistical grouping into three levels – high, medium, low. At the second stage, a representative sample of highly profitable and low-profit agricultural enterprises of the Saratov region was drawn up, which served as the basis for the development of the method. In the third stage, a retrospective analysis of the analyzed indicators for selected enterprises was conducted, based on which their transition probabilities and mean values ​​were calculated in the fourth and fifth stages. In the sixth and seventh stages, based on the values ​​of transition probabilities and average values, short-term and long-term scenario forecasts of the profitability level of selected enterprises were compiled separately for each group. On an exclusive stage, on the basis of pessimistic, realistic and optimistic scenarios, alternative management solutions aimed at stimulating production processes are proposed.

Model predictive control under forecast uncertainty for optimal operation of buildings with integrated solar systems

In this paper, we explore intelligent operation strategies, based on stochastic model predictive control (SMPC), for optimal utilization of solar energy in buildings with integrated solar systems. Our approach takes into account the uncertainty in solar irradiance forecast over a prediction horizon, using a new probabilistic time series autoregressive model, calibrated on the sky-cover forecast from a weather service provider. In the optimal control formulation, we model the effect of solar irradiance as non-Gaussian stochastic disturbance affecting the cost and constraints, and the nonconvex cost function is an expectation over the stochastic process. To solve this complex optimization problem, we introduce a new approximate dynamic programming methodology that represents the optimal cost-to-go functions using Gaussian process regression, and achieves good solution quality. In the final step, we use an emulator that couples physical system models in TRNSYS with the SMPC controller developed using Python and MATLAB to evaluate the closed-loop operation of a building-integrated system with a solar-assisted heat pump coupled with radiant floor heating. For the system and climate under consideration, the SMPC saves up to 44% of the electricity consumption for heating in a winter month, compared to a baseline well-tuned rule-based controller, and it is robust, imposing less uncertainty on thermal comfort violation.

Approximate dynamic programming algorithms for multidimensional flexible production-inventory problems

An important issue in the manufacturing and supply chain literature concerns the optimisation of inventory decisions. Single-product inventory problems are widely studied and have been optimally solved under a variety of assumptions and settings. However, as systems become more complex, inventory decisions become more complicated for which the methods/approaches for optimising single inventory systems are incapable of deriving optimal policies. Manufacturing process flexibility provides an example of such a complex application area. Decisions involving the interrelated product inventories and production facilities form a highly multidimensional, non-decomposable system for which optimal policies cannot be readily obtained. We propose the methodology of approximate dynamic programming (ADP) to overcome the computational challenge imposed by this multidimensionality. Incorporating a sample backup simulation approach, ADP develops policies by utilising only a fraction of the computations required by classical dynamic programming. However, there are few studies in the literature that optimise production decisions in a stochastic, multi-factory, multi-product inventory system of this complexity. This paper aims to explore the feasibility and relevancy of ADP algorithms for this application. We present the results from numerical experiments that establish the strong performance of policies developed via temporal difference ADP algorithms in comparison to optimal policies and to policies derived from a deterministic approximation of the problem.