Traditional Dynamic Programming Research Articles

Ascent trajectory tracking method using time-varying quadratic adaptive dynamic programming

Ascent trajectory tracking in the longitudinal plane is a class of nonaffine noncascade nonlinear system control problems with single input and multiple output, which is difficult to control by nonlinear method directly. Adaptive dynamic programming algorithm has the advantages of precise control and adaptability for general nonlinear control problem, and the time-varying quadratic adaptive dynamic programming algorithm proposed in this paper promotes the convergence and calculation speed of the traditional adaptive dynamic programming algorithm. To implement the algorithm effectively, the independent variables of ascent model are substituted in the launching coordinate, and then the model is treated as a discrete nominal trajectory tracking problem. Besides, the heuristic dynamic programming structure is used to train the processed model, and thus only the time-varying weight in the designed evaluation network needs to be updated. Simulation shows that the proposed algorithm can update the control variable online after predicting the cost function offline with the dynamic equations, which is faster than the general adaptive dynamic programming algorithm. In addition, the proposed algorithm can effectively and accurately track the nominal trajectory under the uncertainty of parameters compared with the linear quadratic regulators algorithm.

A Computationally Efficient Optimization Approach for Battery Systems in Islanded Microgrid

In islanded microgrids, it is a challenge to optimize battery energy storage systems (BESSs) with other power supply units (e.g., renewable energy and traditional power generator) and achieve the minimum daily operational cost. In this paper, we propose a computationally efficient near optimal control approach to tackle this problem. Specifically, a new islanded microgrid model, including the power supply and demand as well as battery lifetime characteristics, has been formulated based on Markov decision process. Then, we propose an approximate dynamic programming approach to solve this energy optimization problem, and achieve near minimum operational cost efficiently. We use linear programming (LP) and dynamic programming (DP) approaches to validate the percentage of optimality of our proposed approach for deterministic and stochastic case studies, respectively. Our proposed approach can obtain 100% optimality in deterministic cases comparing with results from LP, and can obtain competitive percentages of optimality with around 50% less computational time in stochastic cases comparing with results from DP. Moreover, we quantify that discharging the BESS at a certain state of charge range can increase the battery lifetime and the yearly net savings of the system. In addition, we validate our proposed approach for different sets of large data samples and achieve 18.69 times faster response than that of the traditional DP approach for 0.5 million of data samples.

Approximate Dynamic Programming Based Control of Proppant Concentration in Hydraulic Fracturing

Hydraulic fracturing has played a crucial role in enhancing the extraction of oil and gas from deep underground sources. The two main objectives of hydraulic fracturing are to produce fractures with a desired fracture geometry and to achieve the target proppant concentration inside the fracture. Recently, some efforts have been made to accomplish these objectives by the model predictive control (MPC) theory based on the assumption that the rock mechanical properties such as the Young’s modulus are known and spatially homogenous. However, this approach may not be optimal if there is an uncertainty in the rock mechanical properties. Furthermore, the computational requirements associated with the MPC approach to calculate the control moves at each sampling time can be significantly high when the underlying process dynamics is described by a nonlinear large-scale system. To address these issues, the current work proposes an approximate dynamic programming (ADP) based approach for the closed-loop control of hydraulic fracturing to achieve the target proppant concentration at the end of pumping. ADP is a model-based control technique which combines a high-fidelity simulation and function approximator to alleviate the “curse-of-dimensionality” associated with the traditional dynamic programming (DP) approach. A series of simulations results is provided to demonstrate the performance of the ADP-based controller in achieving the target proppant concentration at the end of pumping at a fraction of the computational cost required by MPC while handling the uncertainty in the Young’s modulus of the rock formation.

Semantic Edge Based Disparity Estimation Using Adaptive Dynamic Programming for Binocular Sensors.

Disparity calculation is crucial for binocular sensor ranging. The disparity estimation based on edges is an important branch in the research of sparse stereo matching and plays an important role in visual navigation. In this paper, we propose a robust sparse stereo matching method based on the semantic edges. Some simple matching costs are used first, and then a novel adaptive dynamic programming algorithm is proposed to obtain optimal solutions. This algorithm makes use of the disparity or semantic consistency constraint between the stereo images to adaptively search parameters, which can improve the robustness of our method. The proposed method is compared quantitatively and qualitatively with the traditional dynamic programming method, some dense stereo matching methods, and the advanced edge-based method respectively. Experiments show that our method can provide superior performance on the above comparison.

Towards Cost and Comfort Based Hybrid Optimization for Residential Load Scheduling in a Smart Grid

In a smart grid, several optimization techniques have been developed to schedule load in the residential area. Most of these techniques aim at minimizing the energy consumption cost and the comfort of electricity consumer. Conversely, maintaining a balance between two conflicting objectives: energy consumption cost and user comfort is still a challenging task. Therefore, in this paper, we aim to minimize the electricity cost and user discomfort while taking into account the peak energy consumption. In this regard, we implement and analyse the performance of a traditional dynamic programming (DP) technique and two heuristic optimization techniques: genetic algorithm (GA) and binary particle swarm optimization (BPSO) for residential load management. Based on these techniques, we propose a hybrid scheme named GAPSO for residential load scheduling, so as to optimize the desired objective function. In order to alleviate the complexity of the problem, the multi dimensional knapsack is used to ensure that the load of electricity consumer will not escalate during peak hours. The proposed model is evaluated based on two pricing schemes: day-ahead and critical peak pricing for single and multiple days. Furthermore, feasible regions are calculated and analysed to develop a relationship between power consumption, electricity cost, and user discomfort. The simulation results are compared with GA, BPSO and DP, and validate that the proposed hybrid scheme reflects substantial savings in electricity bills with minimum user discomfort. Moreover, results also show a phenomenal reduction in peak power consumption.

New Algorithm for Determining the Similarities of Nucleotide Sequences Based on Comparisons of Torsion of Discrete Curves

Discrete curves can be used to characterize sequences and the complete coding information of DNA in three-dimensional space. When the curve possesses properties that, in a mapping relationship, mirror those of DNA sequences, its characteristic values reveal the biological characteristics of a DNA sequence. By describing a DNA sequence using the recent Z curve model, we here propose a new method to judge the similarities of discrete DNA curves based on comparisons of the torsion of the curves. This comparative method was verified by quantitatively analyzing the similarities of cDNA sequences of 10 types of avian influenza viruses. Compared with a traditional dynamic programming algorithm, this new method takes full advantage of computer geometry for genetic comparative analyses by shifting from the traditional approach of a direct comparison of gene sequence characteristics to a comparison of the similarity of spatially discrete curves.

The Hydropower Station Output Function and its Application in Reservoir Operation

Dynamic programming(DP) is an effective and powerful mathematical tool to solve reservoir operation optimization(ROO) problems because it can yield global optimal solutions. But with the increase of the reservoirs’ number, DP will face problems like long computation time, large calculation scale, which is called ‘curse of dimensionality’. Heuristic random search algorithms and improved DP algorithms can decrease the computation time but they can only get near-optimal solutions. By analyzing the principle of DP in ROO, this paper finds that DP involves complex iterative computation and repeated procedures, which is very time-consuming. Aiming at solving these problems, this paper proposes a new notion called Hydropower Station Output Function (HSOF) and uses it to optimize and improve traditional DP, which can reduce the iterative computation and repeated procedures then improve computational efficiency. This paper takes Yayangshan hydroelectric power station in Li Xianjiang River Basin and the cascade hydropower station consisting of Yayangshan and Shimenkan hydropower stations as examples for single reservoir optimal operation (SROO) and cascade reservoir operation optimization (CROO). By comparing with the operation result of traditional DP and the progressive optimality algorithm (POA), case presents that the improved DP based on HSOF can reduce the programming complexity of DP, thus effectively alleviates the time-consuming problem and in the meantime, keeps the global convergence feature of DP.

HEV power management control strategy for urban driving

The aim of this paper is to further improve the fuel efficiency and keep the balance of battery SOC simultaneously during urban driving. A dynamic programming (DP)-optimized HEV power management control strategy is proposed. The DP algorithm is applied to obtain the optimal engine/motor power distribution and utilized for the design of the fuzzy control strategy. The traditional DP algorithm is modified with the consideration of SOC balance for HEVs. In the analysis of DP simulation results, rules of torque slip behaviors have been found, which are directly utilized in the design of fuzzy control strategy. In order to improve the practicality of the control strategy to meet the diversities of city driving patterns, an urban driving pattern recognition method is presented. To evaluate the control performance, the proposed control strategy is also compared with the conventional rule-based strategy. The simulation results indicate that by adopting the proposed strategy the fuel efficiency of HEV is improved, and the SOC of the battery is kept in balance during different urban driving cycles.

Elliptic Shape Prior Dynamic Programming for Accurate Vessel Segmentation in MRI Sequences with Automated Optimal Parameter Selection

This study proposes an accurate vessel segmentation method using an elliptic-model-guided dynamic programming algorithm to find the shortest path in a two-dimensional matrix. The elliptic model increases the algorithm’s resistance to noise around the vessel boundaries. To test the system reliability and accuracy, we use phantom images with added uniformly distributed noise to get different signal-to-noise ratios. In addition, the optimal parameter values are determined via a phantom study. We further apply this method to detect the boundaries of the common carotid artery (CCA) in real magnetic resonance imaging (MRI) sequences without contrast agent injection. Manual tracings of the CCA boundaries are performed by well-trained experts as the gold standard. Comparisons between the manual tracings and automated results are made on 8 MRI sequences (400 total images). The average unsigned error rate is 3.6 % (standard deviation = 2.4 %). The results demonstrate that the proposed method is qualitatively better than traditional dynamic programming for vessel boundary detection on MRI sequences without contrast agent injection.

A Pseudospectral Strategy for Optimal Power Management in Series Hybrid Electric Powertrains

This paper examines the problem of optimizing hybrid electric vehicle (HEV) power management for fuel economy. This paper begins by presenting a pseudospectral algorithm to solve this optimization problem. Compared with traditional dynamic programming (DP)-based optimal power management approaches, this algorithm has two key advantages: It is numerically more efficient, and it furnishes both the optimal state and costate trajectories. Building on the second advantage, this paper proposes a two-level strategy for optimal power management in vehicles commuting along fixed routes. The upper level of the proposed strategy is a costate adaptation algorithm employing pseudospectral optimization, whereas the lower level is an instantaneous optimization controller employing Pontryagin's minimum principle (PMP). This paper shows its pseudospectral optimization algorithm and two-level strategy using numerical simulation for a series hybrid school bus. Parameters of the bus powertrain model are obtained from experimental component tests performed at the National Engineering Laboratory for Electric Vehicles, Beijing Institute of Technology. Simulation results show that the pseudospectral method reaches a solution close to DP with higher computational efficiency. Furthermore, the proposed two-level strategy is capable of adapting vehicle power management based on road-grade predictions, i.e., an attractive feature compared with more traditional online hybrid power management approaches such as the use of proportional integral derivative (PID) control for adaptive equivalent fuel consumption minimization [PID equivalent consumption minimization strategy (PID-ECMS)].

Stereo matching using dynamic programming based on differential smoothing

The traditional dynamic programming considers little about the smoothness at edges, often causing the mismatching of disparity in depth-discontinuous regions. To solve this problem, stereo matching using dynamic programming based on differential equation is proposed. This method exploits all of the matching cost function, the smooth energy function and the occlusion cost function. Given the difference of adjacent pixels in smooth and edge regions might be varied, a smooth energy function is proposed. Diffusion velocity of this energy function incorporates gradient information, line points as well as corner points, thereby reducing mismatching rate in depth-discontinuous regions. Several experiments have been done to prove that, stereo matching using both of dynamic programming and differential smoothing could reduce mismatching rate of disparity by a large margin.

Physical topology optimization of infrastructure health monitoring sensor network for high-speed rail

With the rapid development of high-speed rail (HSR) systems, the security and safety of these huge systems are becoming the primary concerns for passengers. HSR infrastructure plays an important role in HSR systems, making the maintenance of security and safety of the HSR infrastructure especially important. Meanwhile, sensor network technologies allow the realization of real-time and all-weather monitoring of HSR infrastructure. This paper analyzes the application requirements and characteristics of infrastructure health monitoring sensor network (IHMSN) through construction of a three-layer IHMSN which is composed of end devices, repeater points, and access points. The physical topology optimization goal of IHMSN is to set the optimal number of network nodes (namely, minimum cost) as well as the best physical connections. Given types and amount of the end devices, a multiple knapsack model is established which converts the physical topology optimization problems into multiple knapsack problems. Based on the different needs of practical application, three different cases (basis case, adding devices case and weight-based case) are proposed, and the corresponding models are built. Some artificial intelligence algorithms and a traditional dynamic programming algorithm are presented to solve the problems. In addition, a general algorithmic finite state machine is proposed to describe the solving process. After comparing these algorithms in execution time, memory, and optimal results, the genetic algorithm and particle swarm optimization algorithm stand out when used to solve the basic case as well as the extended cases. The numerical results show that these proposed models and algorithms can effectively solve the physical topology optimization problem of IHMSN for HSR systems. Moreover, these methods can effectively reduce network costs and provide a theoretical basis for network communication link optimization.

Global and local sequence alignment with a bounded number of gaps

Pairwise sequence alignment techniques have gained renewed interest in recent years, primarily due to their applications in re-sequencing—the assembly of a genome directed by a reference sequence.In this article, we show that adding the flexibility of bounding the number of gaps inserted in an alignment strengthens the classical sequence alignment scheme of scoring matrices and affine gap penalty scores. We present GapsMis, an algorithm for pairwise global sequence alignment with a variable, but bounded, number of gaps. It is based on computing a variant of the traditional dynamic programming matrix for global sequence alignment. We also present GapsMis-L, the analogous algorithm for pairwise local sequence alignment with a variable, but bounded, number of gaps.To test the accuracy of GapsMis and GapsMis-L we performed millions of pairwise sequence alignments under realistic conditions, based on the properties of real full-length genomes. The results show that GapsMis and GapsMis-L can increase the accuracy of extending short-read alignments compared to the traditional approaches. The importance of our contribution is underlined by the fact that the provided algorithms may be seamlessly integrated into any biological pipeline. The open-source code of our implementation is freely available at http://www.inf.kcl.ac.uk/research/projects/gapmis/.

Approximately Optimal Risk-Averse Routing Policies via Adaptive Discretization

Mitigating risk in decision-making has been a long-standing problem. Due to the mathematical challenge of its nonlinear nature, especially in adaptive decision-making problems, finding optimal policies is typically intractable. With a focus on efficient algorithms, we ask how well we can approximate the optimal policies for the difficult case of general utility models of risk. Little is known about efficient algorithms beyond the very special cases of linear (risk-neutral) and exponential utilities since general utilities are not separable and preclude the use of traditional dynamic programming techniques. In this paper, we consider general utility functions and investigate efficient computation of approximately optimal routing policies, where the goal is to maximize the expected utility of arriving at a destination around a given deadline. We present an adaptive discretization variant of successive approximation which gives an $\error$-optimal policy in polynomial time. The main insight is to perform discretization at the utility level space, which results in a nonuniform discretization of the domain, and applies for any monotone utility function.

Analysis and comparison of optimal power management strategies for a series plug-in hybrid school bus via dynamic programming

This paper examines two optimal power management strategies, i.e., blending (BL) strategy and charge-depleting/charge-sustaining (CD/CS) strategy, for a series plug-in school bus. The minimum fuel consumption results of these two strategies are compared and the pattern of the globally optimal battery state of charge (SoC) trajectories is analysed. Dynamic programming (DP) approach which is able to achieve global optimisation is applied in this study to guarantee a fair comparison. More specifically, to overcome interpolation leakage in traditional DP algorithms, a level-set algorithm is utilised. Simulation results show that: 1) The globally optimal way to control is to linearly deplete the battery SoC to its minimum allowable value when the vehicle reaches to the end of the journey; 2) The difference between the optimal fuel economy that can be achieved by the BL and CD/CS strategies is very marginal for a series plug-in powertrain, which is in the order of 1-2%.

Cycle-based optimal NOx emission control of selective catalytic reduction systems with dynamic programming algorithm

In this work, we investigate the cycle-based optimal NOx emission control for Diesel engine selective catalytic reduction (SCR) systems. The optimization is carried out based on the SCR model and the SCR is assumed to be a continuous stirred tank reactor during the modeling. Considering the main chemical reactions inside an SCR cell, a three-state nonlinear model is obtained and the system parameters are determined by using the experimental data. The states of the nonlinear model consist of the NOx concentration, the ammonia concentration, and the ammonia coverage ratio. Though there are considerable works on the SCR dosing control in the literature, the optimal control strategy applied to the SCR systems has been rarely studied. The global optimality in the SCR dosing control can be used as a benchmark and evaluate the performance of other designed controllers. Moreover, it can be employed to find the limit of NOx conversion efficiency of a specific SCR aftertreatment system and select the minimal volume of the SCR catalyst cell. Therefore, it is quite meaningful to study the optimal NOx emission control of SCR systems via the dynamic programming approach. For the optimization, the challenges arise from the following aspects: (1) Due to the system stiffness, we cannot find an equivalent discrete-time model for the traditional dynamic programming, that is, the traditional dynamic programming algorithm is not applicable. (2) The computational load is relatively too heavy and a personal PC is not affordable. To deal with the existing challenges, we propose the dynamic programming algorithm based on the continuous-time SCR model. The computational load is adjustable in terms of the accuracy. The proposed algorithm is then applied to the SCR system for the NOx emission reduction. And the algorithm validation is carried out based on the US06 test cycle. It infers from the optimization results and the comparison, the prescribed ammonia slip constraint is satisfied and the tailpipe NOx emission is optimized.

Heuristic Method for Identifying Concave Pareto Frontiers in Multi-Objective Dynamic Programming Problems

Aerospace engineers and analysts tasked with informing system-level decisions commonly seek to identify the frontier of Pareto-optimal solutions with respect to objectives of interest. For problems in which decisions are made over multiple stages or periods, dynamic programming can be an efficient and effective method for identifying such a Pareto frontier. To employ traditional dynamic programming, however, a single objective function must be defined. Aggregation of multiple objectives into a single objective using traditional simple additive weighting has the limitation of permitting identification of only points on convex portions of the Pareto frontier. This can translate into detection of a frontier with significant and misleading gaps. This paper proposes a theory-motivated aggregation function modification and method to improve the ability of dynamic programming procedures to detect concave portions of Pareto frontiers in multi-objective, multistage problems. Following a theoretical motivation and method definition, a military aircraft route planning example is provided to illustrate the method’s accuracy and efficiency.

Stereo Matching Using Iterative Dynamic Programming Based on Color Segmentation of Images

The traditional dynamic programming stereo matching algorithms usually adopt the disparity assumption based on the intensity change of images; With the development of stereo matching technique, the disparity assumption based on image color segmentation is proved to meet better the need of true scenes. The paper introduces the disparity assumption into the stereo matching using dynamic programming, proposes a new global energy function, which not only resolves the problem of traditional dynamic programming stereo matching algorthm that the energy function is short of intensity and disparity constraints between scan lines but also can be computed more exactly because the adopted dissimillarity function propsed by Birchfield is extended from 2-connect neighborhood to 8-connect neighborhood. The energy function can converge fast because of the proposed pruning algorithm based on color segmentation. The experiments show that the proposed method produces competitive results with the 2-dimensional energy function minimized algorithm but has the much lower computing cost than them.

A Case Study on Air Combat Decision Using Approximated Dynamic Programming

As a continuous state space problem, air combat is difficult to be resolved by traditional dynamic programming (DP) with discretized state space. The approximated dynamic programming (ADP) approach is studied in this paper to build a high performance decision model for air combat in 1 versus 1 scenario, in which the iterative process for policy improvement is replaced by mass sampling from history trajectories and utility function approximating, leading to high efficiency on policy improvement eventually. A continuous reward function is also constructed to better guide the plane to find its way to “winner” state from any initial situation. According to our experiments, the plane is more offensive when following policy derived from ADP approach other than the baseline Min-Max policy, in which the “time to win” is reduced greatly but the cumulated probability of being killed by enemy is higher. The reason is analyzed in this paper.

Dynamic programming algorithms for the bi-objective integer knapsack problem

This paper presents two new dynamic programming (DP) algorithms to find the exact Pareto frontier for the bi-objective integer knapsack problem. First, a property of the traditional DP algorithm for the multi-objective integer knapsack problem is identified. The first algorithm is developed by directly using the property. The second algorithm is a hybrid DP approach using the concept of the bound sets. The property is used in conjunction with the bound sets. Next, the numerical experiments showed that a promising partial solution can be sometimes discarded if the solutions of the linear relaxation for the subproblem associated with the partial solution are directly used to estimate an upper bound set. It means that the upper bound set is underestimated. Then, an extended upper bound set is proposed on the basis of the set of linear relaxation solutions. The efficiency of the hybrid algorithm is improved by tightening the proposed upper bound set. The numerical results obtained from different types of bi-objective instances show the effectiveness of the proposed approach.