Well-known Dynamic Programming Research Articles

Adaptive optimal control of continuous-time nonlinear affine systems via hybrid iteration

In this paper, a novel successive approximation framework, named hybrid iteration (HI), is proposed to fill up the performance gap between two well-known dynamic programming algorithms, namely policy iteration (PI) and value iteration (VI). Using HI, an approximated optimal control policy can be learned without prior knowledge of an initial admissible control policy required by PI. Additionally, the HI algorithm converges to the optimal solution much faster than VI, and thus requires tremendously less number of learning iterations and CPU-time, compared to VI. Initially, we develop a model-based HI algorithm, and then extend it to a data-driven HI algorithm which learns the optimal control policy without any information of the physics of the system. Simulation results demonstrate the efficacy of the proposed HI algorithm.

Discrete-time switching control in random walks

In this paper we study a discrete-time switching control problem when the dynamic of the system evolves as a random walk. The payoff function is a discounted-type total cost whose discount factor may depend on the state and/or the action variables. This flexibility includes cases when the controller can stop the dynamics of the system whether is optimal to do it. To provide optimality results, we use the well-known dynamic programming method that leads to the study of certain functional equations. Under suitable conditions, we prove the existence of solutions to these equations, and we also show that the optimal value of the control problem becomes a minimal solution of these equations. Finally, we illustrate our results through an example about the control of epidemic processes.

ON ROUTING PROBLEM WITH STARTING POINT OPTIMIZATION

One problem focused on engineering applications is considered. It is assumed that sequential visits to megacities have been implemented. After all visits have been made, it is required to return to the starting point (a more complex dependence on the starting point is also considered). But the last requirement is not strict: some weakening of the return condition is acceptable. Under these assumptions, it is required to optimize the choice of starting point, route, and specific trajectory. The well-known dynamic programming (DP) is used for the solution. But when using DP, significant difficulties arise associated with the dependence of the terminal component of the criterion on the starting point. Starting point enumeration is required. We consider the possibility of reducing the enumeration associated with applied variants of universal (relative to the starting point) dynamic programming. Of course, this approach requires some transformation of the problem.

Depth sensing with disparity space images and three-dimensional recursive search

ABSTRACTWe present a coarse-to-fine stereo matching optimization applicable to methods utilizing the Disparity Space Image (DSI) structure. With the Three-dimensional Recursive Search algorithm (3DRS), a coarse disparity seed is obtained first, with minimal computational effort. The coarse disparity seed is then used as a guidance to locally compute the DSI disparity space with a reduced number of disparity hypotheses, yielding significantly shorter execution times for the disparity computation. The method performance was measured on the well-known Dynamic Programming (DP) DSI-based method and the images from the Middlebury set. The DP method with the DSI optimization applied maintains or improves the overall level of disparity map accuracy while delivering a near sevenfold speed-up of execution in comparison to DP alone. We furthermore show that the optimized method's performance does not depend on the expected input disparity range, which is commonly restricted, or expected to be defined upfront, for DSI-based stereo matching methods.

Dynamic programming for discrete-time finite-horizon optimal switching problems with negative switching costs

Abstract In this paper we study a discrete-time optimal switching problem on a finite horizon. The underlying model has a running reward, terminal reward, and signed (positive and negative) switching costs. Using optimal stopping theory for discrete-parameter stochastic processes, we extend a well-known explicit dynamic programming method for computing the value function and the optimal strategy to the case of signed switching costs.

On scaling dynamic programming problems with a multithreaded tabling Prolog system

Prolog system scaling the execution of dynamic programming problems.New design supporting multithreaded tabling in Prolog systemsApplication of multithreaded tabling to well-known dynamic programming problems.Experimental results showing promising speedup results up to 32 cores. Tabling is a powerful implementation technique that improves the declarativeness and expressiveness of traditional Prolog systems in dealing with recursion and redundant computations. It can be viewed as a natural tool to implement dynamic programming problems, where a general recursive strategy divides a problem in simple sub-problems that are often the same. When tabling is combined with multithreading, we have the best of both worlds, since we can exploit the combination of higher declarative semantics with higher procedural control. However, at the engine level, such combination for dynamic programming problems is very difficult to exploit in order to achieve execution scalability as we increase the number of running threads. In this work, we focus on two well-known dynamic programming problems, the Knapsack and the Longest Common Subsequence problems, and we discuss how we were able to scale their execution by using the multithreaded tabling engine of the Yap Prolog system. To the best of our knowledge, this is the first work showing a Prolog system to be able to scale the execution of multithreaded dynamic programming problems. Our experiments also show that our system can achieve comparable or even better speedup results than other parallel implementations of the same problems.

Adaptive Search and Tracking of Sparse Dynamic Targets Under Resource Constraints

This paper considers the problem of resource-constrained and noise-limited localization and estimation of dynamic targets that are sparsely distributed over a large area. We generalize an existing framework [Bashan , 2008] for adaptive allocation of sensing resources to the dynamic case, accounting for time-varying target behavior such as transitions to neighboring cells and varying amplitudes over a potentially long time horizon. The proposed adaptive sensing policy is driven by minimization of a surrogate function for mean squared error within locations containing targets. We provide theoretical upper bounds on the performance of adaptive sensing policies by analyzing solutions with oracle knowledge of target locations, gaining insight into the effect of target motion and amplitude variation as well as sparsity. Exact minimization of the multistage objective function is infeasible, but myopic optimization yields a closed-form solution. We propose a simple non-myopic extension, the Dynamic Adaptive Resource Allocation Policy (D-ARAP), that allocates a fraction of resources for exploring all locations rather than solely exploiting the current belief state. Our numerical studies indicate that D-ARAP has the following advantages: (a) it is more robust than the myopic policy to noise, missing data, and model mismatch; (b) it performs comparably to well-known approximate dynamic programming solutions but at significantly lower computational complexity; and (c) it improves greatly upon nonadaptive uniform resource allocation in terms of estimation error and probability of detection.

A neural-network-based online optimal control approach for nonlinear robust decentralized stabilization

In this paper, the robust decentralized stabilization of continuous-time uncertain nonlinear systems with multi control stations is developed using a neural network based online optimal control approach. The novelty lies in that the well-known adaptive dynamic programming method is extended to deal with the nonlinear feedback control problem under uncertain and large-scale environment. Through introducing an appropriate bounded function and defining a modified cost function, it can be observed that the decentralized optimal controller of the nominal system can achieve robust decentralized stabilization of original uncertain system. Then, a critic neural network is constructed for solving the modified Hamilton---Jacobi---Bellman equation corresponding to the nominal system in an online fashion. The weights of the critic network are tuned based on the standard steepest descent algorithm with an additional term provided to guarantee the boundedness of system states. The stability analysis of the closed-loop system is carried out via the Lyapunov approach. At last, two simulation examples are given to verify the effectiveness of the present control approach.

Stereo matching for large-scale high-resolution satellite images using new tiling technique

Stereo matching has been grabbing the attention of researchers because it plays an important role in computer vision, remote sensing and photogrammetry. Although most methods perform well with small size images, experiments applying them to large-scale data sets under uncontrolled conditions are still lacking. In this paper, we present an empirical study on stereo matching for large-scale high-resolution satellite images. A new method is studied to solve the problem of huge size and memory requirement when dealing with large-scale high resolution satellite images. Integrating the tiling technique with the well-known dynamic programming and coarse-to-fine pyramid scheme as well as using memory wisely, the suggested method can be utilized for huge stereo satellite images. Analyzing 350 points from an image of size of 8192 x 8192, disparity results attain an acceptable accuracy with RMS error of 0.5459. Taking the trade-off between computational aspect and accuracy, our method gives an efficient stereo matching for huge satellite image files.

A Tutorial on Linear Function Approximators for Dynamic Programming and Reinforcement Learning

A Markov Decision Process (MDP) is a natural framework for formulating sequential decision-making problems under uncertainty. In recent years, researchers have greatly advanced algorithms for learning and acting in MDPs. This article reviews such algorithms, beginning with well-known dynamic programming methods for solving MDPs such as policy iteration and value iteration, then describes approximate dynamic programming methods such as trajectory based value iteration, and finally moves to reinforcement learning methods such as Q-Learning, SARSA, and least-squares policy iteration. We describe algorithms in a unified framework, giving pseudocode together with memory and iteration complexity analysis for each. Empirical evaluations of these techniques with four representations across four domains, provide insight into how these algorithms perform with various feature sets in terms of running time and performance.

A Modified Dynamic Parallel Algorithm for Sequence Alignment in BioSequences

paper presents a Modified Dynamic Parallel Algorithm for Sequence Alignment in BioSequences. DNA sequence alignment between two or more bio sequences using algorithms is a complex problem due to its applicability in the field of biology. This algorithm achieves its efficiency in using computational resources by making a M X N matrix , M represents the length of first sequence and N represents the length of the second sequence .This modified Dynamic algorithm performs calculation to fill the three main diagonal cells without evaluating other cells in the matrix. The proposed model is based on Index Based Pattern Matching using Multithreading (13) to obtain the optimal alignment using Multithreading. The executed results indicate that with the proposed algorithm Memory Efficiency and Fast Execution are achieved over the well-known dynamic programming approach Needleman-Wunsch and Hirschberg approach. Keywordsoperations, sequence alignment, Multithreading, score.

Reliability constrained multi-period generation expansion planning of electrical energy resources using MILP

SUMMARY The main goal of Generation Expansion Planning (GEP) is to minimize total costs associated with new power generating units’ installation subject to technical and economical constraints. This paper addresses the GEP with probabilistic reliability criteria. The Loss-of-Load Probability reliability index is explicitly augmented as a new constraint which takes into account the reserve requirements. The outage cost is represented by the Expected Energy Not Served index. Due to nonlinear nature of these reliability indices, the GEP optimization problem with reliability criteria is very complicated to solve. Accordingly, the focus of this work is to deal with the reliability constrained multi-period GEP problem as a Mixed Integer Linear Programming (MILP). The results in the case study indicate the effect of reliability considerations on decision-making process. The simulation results also show the superiority of the proposed MILP-based method in comparison with the well-known metaheuristic algorithms and Dynamic Programming approach in the viewpoints of the accuracy and computational speed. Copyright © 2012 John Wiley & Sons, Ltd.

An application of infinite horizon stochastic dynamic programming in multi-stage project investment decision-making

In multi-stage project investment decision-making with uncertainty, risk mitigation plays a vital role. The return on investment (ROI) that will be realised in making a particular decision quite often carries a high degree of uncertainty, with an increased number of competing investors entering to the market every day. In this research, our objective is to develop a technique for a multi-stage project investment decision problem that deals with uncertainty in ROI and complex interrelated state transition dynamics. We do this by formulating our problem as an infinite horizon stochastic dynamic programming (IHSDP) problem and solve it to maximise the total return over an infinite time horizon. We have implemented our solution to the project investment decision problem in a simple case study using three well-known stochastic dynamic programming algorithms. Our simulation results show that the IHSDP algorithms are useful in making optimum investment decisions in an uncertain business environment.

Sequence Alignment by Regression Coding

In aligning two sequences, dynamic time warping (DTW) is the well-known dynamic programming algorithm. However, DTW can be sensitive to noise samples that may affect alignment of other relevant samples. In this article, we propose a novel approach to sequence alignment by treating it as a regression coding optimization problem, a task to predict one sequence from another. With some mild relaxation DTW can be seen as a special case of our approach while we provide more flexible and informative interpretation. Experimental results on both synthetic and real-world datasets show that our method can yield more accurate alignment than existing approaches.

Determination of edge correspondence using color codes for one-shot shape acquisition

A robust method for measuring 3D shapes is proposed, in which only one stripe pattern image is required. To determine edge correspondence, we match color codes instead of edge codes because the former are more stable and immune to the standard deviation of the Gaussian filter in edge detection and width of the color band. The color code is identified by K-means. This method exhibits huge advantages in adaptability and automation over thresholding techniques. The proposed decoding method is compared with two well-known algorithms, dynamic programming and multi-pass dynamic programming. Using ground truth, we evaluate the performance of the methods in measuring three different objects. Quantitative and qualitative comparisons are shown in the experiments, and results affirm that our method is effective and robust.

A distortion optimal rate allocation algorithm for transmission of embedded bitstreams over noisy channels

We propose a distortion optimal rate allocation algorithm for robust transmission of embedded bitstreams over noisy channels. The algorithm is based on the backward application of a Viterbi-like algorithm to a search trellis, and can be applied to both scenarios of fixed and variable channel packet length problems, referred to as FPP and VPP, respectively. For the VPP, the complexity of the algorithm is comparable to the well-known dynamic programming approach of Chande and Farvardin. For the FPP, where no low-complexity algorithm is known, the complexity of the proposed algorithm is O(N/sup 2/), where N is the number of transmitted packets.

Speeding up the Dreyfus–Wagner algorithm for minimum Steiner trees

The Dreyfus–Wagner algorithm is a well-known dynamic programming method for computing minimum Steiner trees in general weighted graphs in time O *(3 k ), where k is the number of terminal nodes to be connected. We improve its running time to O *(2.684 k ) by showing that the optimum Steiner tree T can be partitioned into T = T 1∪ T 2 ∪ T 3 in a certain way such that each T i is a minimum Steiner tree in a suitable contracted graph G i with less than $${\frac{k}{2}}$$ terminals. In the rectilinear case, there exists a variant of the dynamic programming method that runs in O *(2.386 k ). In this case, our splitting technique yields an improvement to O *(2.335 k ).

BIT-PARALLEL COMPUTATION OF LOCAL SIMILARITY SCORE MATRICES WITH UNITARY WEIGHTS

Local similarity computation between two sequences permits detecting all the relevant alignments present between subsequences thereof. A well-known dynamic programming algorithm works in time O(mn), m and n being the lengths of the subsequences. The algorithm is rather slow when applied over many sequence pairs. In this paper we present the first bit-parallel computation of the score matrix, for a simplified choice of scores. If the computer word has w bits, then the resulting algorithm works in O(mn log min (m, n, w)/w) time, achieving up to 8-fold speedups in practice. Some DNA comparison applications use precisely the simplified scores we handle, and thus our algorithm is directly applicable. In others, our method could be used as a raw filter to discard most of the strings, so the classical algorithm can be focused only on the substring pairs that can yield relevant results.

Genetic algorithm and Hopfield neural network for a dynamic lot sizing problem

The dynamic lot sizing plays an important role when the demand of an item varies with time. This paper addresses a dynamic lot sizing problem (DLSP) with capacity constraint and discount price structure. Although the well-known dynamic programming (DP) of Wagner-Within is capable of providing an optimal solution for single stage lot sizing problems, it suffers from its high computational complexity. This limits the use of DP in practical problems that are generally larger in size. Two meta-heuristics, genetic algorithm (GA) and Hopfield neural network (HNN) are designed for DLSP to get best trade-off between solution quality and computational time. The DP algorithm is modeled to derive the optimal solution to the experimental problem. The optimality of GA and HNN are tested by comparing the percentage deviation of GA and HNN results against the optimal solution derived using DP.

Matching for Run-Length Encoded Strings

Measuring the similarity between two strings, through such standard measures as Hamming distance, edit distance, and longest common subsequence, is one of the fundamental problems in pattern matching. We consider the problem of finding the longest common subsequence of two strings. A well-known dynamic programming algorithm computes the longest common subsequence of strings X and Y in O(|X|/spl middot/|Y|) time. We develop significantly faster algorithms for a special class of strings which emerge frequently in pattern matching problems. A string S is run-length encoded if it is described as an ordered sequence of pairs (/spl sigma/,i), each consisting of an alphabet symbol /spl sigma/ and an integer i. Each pair corresponds to a run in S consisting of i consecutive occurrences of /spl sigma/. For example, the string aaaabbbbcccabbbbcc can be encoded as a/sup 4/b/sup 4/c/sup 3/a/sup 1/b/sup 4/c/sup 2/. Such a run-length encoded string can be significantly shorter than the expanded string representation. Indeed, runlength coding serves as a popular image compression technique, since many classes of images, such as binary images in facsimile transmission, typically contain large patches of identically-valued pixels.