Dynamic Programming Algorithm Research Articles

Optimal Voltage Recovery Learning Control for Microgrids with N-Distributed Generations via Hybrid Iteration Algorithm

Considering that the nonlinearity and uncertainty of the microgrid model complicate the derivation and design of the optimal controller, an adaptive dynamic programming (ADP) algorithm is designed to solve the model-free non-zero-sum game. By combining the advantages of policy iteration and value iteration, an optimal learning control scheme based on hybrid iteration is constructed to provide stringent real power sharing for the nonlinear and coupled microgrid systems with N-distributed generations. First, using non-zero-sum differential game strategy, a novel distributed secondary voltage recovery consensus optimal control protocol is built using a hybrid iteration method to realize the voltage recovery of microgrids. Then, the data of the system state and input are gathered along a dynamic system trajectory and a data-driven optimal controller learns the game solution without microgrid physics information, enhancing convenience and efficiency in practical applications. Furthermore, the convergence analysis is given in detail, and it is proved that the control protocol can converge to the optimal solution so as to ensure the stability of the voltage recovery of the microgrid system. Convergence analysis proves the convergence of the the protocol to the optimal solution, ensuring voltage recovery stability. Simulation results validate the feasibility and effectiveness of the proposed scheme.

Zero‐sum game‐based H∞ optimal finite‐time prescribed performance control for nonlinear multiagent systems with actuator faults

AbstractThis article proposes an optimal leader‐follower consensus control protocol with finite‐time prescribed performance for nonlinear multiagent systems suffering from the actuator fault based on the zero‐sum game framework. First, to guarantee the specified performance of the system within a finite time frame, a novel error transformation function (ETF) is constructed. The ETF proposed in this paper can adjust the mapping range of error transformation to achieve better error convergence performance, thereby enhancing the flexibility of the controller design. Then, to compensate for the negative impact of actuator fault, an error‐related item is separated from the optimal performance index function to design an online fault estimator, which can obtain a better error estimation accuracy. Furthermore, to lessen the need for difficult‐to‐verify persistent excitation signal in adaptive dynamic programming algorithm, both historical and real‐time data are applied to adjust the updating law of neural network weight. Moreover, it demonstrates that the signals within the closed‐loop system are bounded and the expected finite‐time prescribed performance can be ensured in the presence of faults. Finally, a numerical example of multiple single‐link robots manipulator system is executed to validate the efficacy of the developed scheme.

Logistics vehicle routing optimisation with synchronised transfer

This paper presents a synchronized transfer strategy in which operators can schedule logistics vehicles among employees and arrange the transfer of commodities. The transfer time consists of the vehicle waiting time and commodity transition time. By adopting the strategy, logistics vehicles can visit fewer communities, resulting in savings in transportation cost under ensuring punctual delivery. An integer programming model based on a space-time-state network is proposed to describe the complex transfer process. To simplify the model, time window and vehicle capacity constraints are embedded into the network. The alternating direction method of multipliers (ADMM) is designed to solve the model. In the ADMM-based solution framework, the original model can be converted into a series of shortest-path searching subproblems and iteratively solved using a dynamic programming (DP) algorithm. Our computational results show that the proposed model and algorithm are efficient and competitive.

Constrained Parameterized Differential Dynamic Programming for Waypoint-Trajectory Optimization

Unmanned aerial vehicles (UAVs) are required to pass through multiple important waypoints as quickly as possible in courier delivery, enemy reconnaissance and other tasks to eventually reach the target position. There are two important problems to be solved in such tasks: constraining the trajectory to pass through intermediate waypoints and optimizing the flight time between these waypoints. A constrained parameterized differential dynamic programming (C-PDDP) algorithm is proposed for meeting multiple waypoint constraints and free-time constraints between waypoints to deal with these two issues. By considering the intermediate waypoint constraints as a kind of path state constraint, the penalty function method is adopted to constrain the trajectory to pass through the waypoints. For the free-time constraints, the flight times between waypoints are converted into time-invariant parameters and updated at the trajectory instants corresponding to the waypoints. The effectiveness of the proposed C-PDDP algorithm under waypoint constraints and free-time constraints is verified through numerical simulations of the UAV multi-point reconnaissance problem with five different waypoints. After comparing the proposed algorithm with fixed-time constrained DDP (C-DDP), it is found that C-PDDP can optimize the flight time of the trajectory with three segments to 7.35 s, 9.50 s and 6.71 s, respectively. In addition, the maximum error of the optimized trajectory waypoints of the C-PDDP algorithm is 1.06 m, which is much smaller than that (7 m) of the C-DDP algorithm used for comparison. A total of 500 Monte Carlo tests were simulated to demonstrate how the proposed algorithm remains robust to random initial guesses.

Data‐driven adaptive optimal control for discrete‐time periodic systems

AbstractIn this paper, a problem of data‐driven optimal control is studied for discrete‐time periodic systems with unknown system matrices and input matrices. For this problem, a value iteration‐based adaptive dynamic programming algorithm is proposed to obtain the suboptimal controller. The core of the algorithm proposed in this paper is to obtain an approximation of the unique positive definite solution of the algebraic Riccati equation and the optimal feedback gain matrix by using the collected real‐time data of the system states and control inputs. Without an initial stabilizing feedback gain, the proposed algorithm could be activated by an arbitrary bounded control input. Finally, the effectiveness of the proposed approach is demonstrated by two examples.

Adaptive event‐triggered control and observer design for discrete‐time nonlinear Markov jump systems with DoS attacks using policy iteration‐based adaptive dynamic programming

AbstractThis article addresses adaptive event‐triggered discrete‐time nonlinear Markov jump systems (MJs) with DoS attacks, where the introduced DoS attacks are considered as more general stochastic models with fixed trigger frequency and duration. To solve the optimal control problem, we use an adaptive dynamic programming (ADP) algorithm based on policy iteration (PI). The approximate process is as follows: the performance index function (PIF) is first updated by the iteration policy in advance, and the control policy is obtained from the PIF. Subsequently, an approximate estimation of the optimal PIF and the optimal control policy is made using the actor‐critic structure obtained through neural network techniques. In order to reduce the occupied communication resources required for control policy iteration, we introduce an adaptive event triggering mechanism with an adaptive triggering threshold, which reduces the conservatism of resource occupation by the PIF compared to the fixed‐threshold ETM. In addition, an observer identifying the unknown dynamics part of the system is designed. Finally, using the Lyapunov function, it is shown that the designed control policy ensures the stability and convergence of the MJS, and the designed observer is effective. Simulation examples are given to verify the feasibility of the controller and the observer.

Real-Time Co-optimization of Gear Shifting and Engine Torque for Predictive Cruise Control of Heavy-Duty Trucks

Fuel consumption is one of the main concerns for heavy-duty trucks. Predictive cruise control (PCC) provides an intriguing opportunity to reduce fuel consumption by using the upcoming road information. In this study, a real-time implementable PCC, which simultaneously optimizes engine torque and gear shifting, is proposed for heavy-duty trucks. To minimize fuel consumption, the problem of the PCC is formulated as a nonlinear model predictive control (MPC), in which the upcoming road elevation information is used. Finding the solution of the nonlinear MPC is time consuming; thus, a real-time implementable solver is developed based on Pontryagin’s maximum principle and indirect shooting method. Dynamic programming (DP) algorithm, as a global optimization algorithm, is used as a performance benchmark for the proposed solver. Simulation, hardware-in-the-loop and real-truck experiments are conducted to verify the performance of the proposed controller. The results demonstrate that the MPC-based solution performs nearly as well as the DP-based solution, with less than 1% deviation for testing roads. Moreover, the proposed co-optimization controller is implementable in a real-truck, and the proposed MPC-based PCC algorithm achieves a fuel-saving rate of 7.9% without compromising the truck’s travel time.

Polymorphic dynamic programming by algebraic shortcut fusion

Dynamic programming (DP) is a broadly applicable algorithmic design paradigm for the efficient, exact solution of otherwise intractable, combinatorial problems. However, the design of such algorithms is often presented informally in an ad-hoc manner. It is sometimes difficult to justify the correctness of these DP algorithms. To address this issue, this paper presents a rigorous algebraic formalism for systematically deriving DP algorithms, based on semiring polymorphism. We start with a specification, construct a (brute-force) algorithm to compute the required solution which is self-evidently correct because it exhaustively generates and evaluates all possible solutions meeting the specification. We then derive, primarily through the use of shortcut fusion, an implementation of this algorithm which is both efficient and correct. We also demonstrate how, with the use of semiring lifting, the specification can be augmented with combinatorial constraints and through semiring lifting, show how these constraints can also be fused with the derived algorithm. This paper furthermore demonstrates how existing DP algorithms for a given combinatorial problem can be abstracted from their original context and re-purposed to solve other combinatorial problems. This approach can be applied to the full scope of combinatorial problems expressible in terms of semirings. This includes, for example: optimization, optimal probability and Viterbi decoding, probabilistic marginalization, logical inference, fuzzy sets, differentiable softmax, and relational and provenance queries. The approach, building on many ideas from the existing literature on constructive algorithmics, exploits generic properties of (semiring) polymorphic functions, tupling and formal sums (lifting), and algebraic simplifications arising from constraint algebras. We demonstrate the effectiveness of this formalism for some example applications arising in signal processing, bioinformatics and reliability engineering. Python software implementing these algorithms can be downloaded from: http://www.maxlittle.net/software/dppolyalg.zip.

Lower time bounds for parallel solving of the subset sum problem by a dynamic programming algorithm

SummaryIn the paper, we compute some lower bounds on time of parallel solving of the subset sum problem on a big number of processors by several versions of dynamic programming algorithm Balsub proposed before by Pisinger. Based on these lower bounds, we propose a version of Balsub which could be possibly effectively parallelized.

Research on energy management strategy for fuel cell hybrid electric vehicles based on improved dynamic programming and air supply optimization

It is crucial to accurately calculate the cost function of the energy management strategy (EMS) of the hybrid powertrain to improve the hydrogen economy of the system. This paper proposes an EMS for fuel cell hybrid electric vehicles (FCHEV) based on improved dynamic programming (DP) and air supply optimization to improve economy and reliability. Taking the maximum net power output of the FC system as the target, the optimal oxygen excess ratio (OER) and cathode pressure of the FC system under different current densities are solved by using PSO. A velocity prediction method based on Bi-LSTM is developed to predict short-term velocity changes in real time. The DP algorithm is introduced and the EMS of the DP algorithm based on short-term velocity prediction is developed for real-time hybrid powertrain optimization and management. Based on the results of energy allocation and optimal gas supply conditions of FCs, the cost function of EMS is modified to reallocate the power of the FC system and battery. The results demonstrate that the proposed method achieves the lowest hydrogen consumption compared to the other two algorithms. Remarkably, it reduces the fuel cost by up to 8.85 % compared to the commonly used online DP algorithm.

ESKEMAP: exact sketch-based read mapping

BackgroundGiven a sequencing read, the broad goal of read mapping is to find the location(s) in the reference genome that have a “similar sequence”. Traditionally, “similar sequence” was defined as having a high alignment score and read mappers were viewed as heuristic solutions to this well-defined problem. For sketch-based mappers, however, there has not been a problem formulation to capture what problem an exact sketch-based mapping algorithm should solve. Moreover, there is no sketch-based method that can find all possible mapping positions for a read above a certain score threshold.ResultsIn this paper, we formulate the problem of read mapping at the level of sequence sketches. We give an exact dynamic programming algorithm that finds all hits above a given similarity threshold. It runs in O(|t|+|p|+ℓ2)\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\mathcal {O} (|t| + |p| + \\ell ^2)$$\\end{document} time and O(ℓlogℓ)\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\mathcal {O} (\\ell \\log \\ell )$$\\end{document} space, where |t| is the number of k\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$k$$\\end{document}-mers inside the sketch of the reference, |p| is the number of k\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$k$$\\end{document}-mers inside the read’s sketch and ℓ\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\ell$$\\end{document} is the number of times that k\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$k$$\\end{document}-mers from the pattern sketch occur in the sketch of the text. We evaluate our algorithm’s performance in mapping long reads to the T2T assembly of human chromosome Y, where ampliconic regions make it desirable to find all good mapping positions. For an equivalent level of precision as minimap2, the recall of our algorithm is 0.88, compared to only 0.76 of minimap2.

Evolution-guided value iteration for optimal tracking control

In this article, an evolution-guided value iteration (EGVI) algorithm is established to address optimal tracking problems for nonlinear nonaffine systems. Conventional adaptive dynamic programming algorithms rely on gradient information to improve the policy, which adheres to the first order necessity condition. Nonetheless, these methods encounter limitations when gradient information is intricate or system dynamics lack differentiability. In response to this challenge, evolutionary computation is leveraged by EGVI to search for the optimal policy without requiring an action network. The competition within the policy population serves as the driving force for policy improvement. Therefore, EGVI can effectively handle complex and non-differentiable systems. Additionally, this innovative method has the potential to enhance exploration efficiency and bolster the robustness of algorithms due to its population-based characteristics. Furthermore, the convergence of the algorithm and the stability of the policy are investigated based on the EGVI framework. Finally, the effectiveness of the established method is comprehensively demonstrated through two simulation experiments.

Scheduling with a weight-modifying activity to minimize the total weighted completion time

This paper considers a single-machine scheduling problem to minimize the total weighted completion time with a weight modifying activity, after which the job weights are discounted by a given factor. The problem is known to be ordinary NP-hard. We propose two mixed integer linear programs (MILPs) and a dynamic programming algorithm to optimally solve the problem. Optimality properties are established and then formulated as pruning constraints to improve the problem-solving efficiency of the MILPs. Special cases are discussed and shown to be solvable by polynomial time algorithms. Complexity status of the studied problem with several instance characteristics is shown. Computational experiments indicate that the optimality properties can reduce the computing efforts and that one of the proposed MILPs can solve instances of 200 jobs in a few seconds.

Neural network-based adaptive critic control for saturated nonlinear systems with full state constraints via a novel event-triggered mechanism

This paper investigates the issue of event-triggered optimal control for saturated nonlinear systems with full state constraints. A smooth function is defined to map the constrained states, then the considered system can be transformed into a state unconstrained system with unknown approximate errors. In addition, a novel event-triggered mechanism is proposed via an adaptive dynamic programming algorithm, which can obtain the optimal control law without constructing the so-called event-triggered HJB equation. Moreover, in order to conquer the difficulty caused by the persistence of excitation conditions, the experience replay technique is utilized to design the critic update law. Meanwhile, it is demonstrated that all signals are uniformly ultimately bounded. Finally, two simulation examples are given to demonstrate the effectiveness of the control strategy.

Large-W limit of the knapsack problem.

We formulate the knapsack problem (KP) as a statistical physics system and compute the corresponding partition function as an integral in the complex plane. The introduced formalism allows us to derive three statistical-physics-based algorithms for the KP: one based on the recursive definition of the exact partition function, another based on the large weight limit of that partition function, and a final one based on the zero-temperature limit of the second. Comparing the performances of the algorithms, we find that they do not consistently outperform (in terms of runtime and accuracy) dynamic programming, annealing, or standard greedy algorithms. However, the exact partition function is shown to reproduce the dynamic programming solution to the KP, and the zero-temperature algorithm is shown to produce a greedy solution. Therefore, although dynamic programming and greedy solutions to the KP are conceptually distinct, a statistical physics formalism introduced reveals that the large weight-constraint limit of the former leads to the latter. We conclude by discussing how to extend this formalism in order to obtain more accurate versions of the introduced algorithms and other similar combinatorial optimization problems.

Synesth: Comprehensive Syntenic Reconciliation with Unsampled Lineages

We present Synesth, the most comprehensive and flexible tool for tree reconciliation that allows for events on syntenies (i.e., on sets of multiple genes), including duplications, transfers, fissions, and transient events going through unsampled species. This model allows for building histories that explicate the inconsistencies between a synteny tree and its associated species tree. We examine the combinatorial properties of this extended reconciliation model and study various associated parsimony problems. First, the infinite set of explicatory histories is reduced to a finite but exponential set of Pareto-optimal histories (in terms of counts of each event type), then to a polynomial set of Pareto-optimal event count vectors, and this eventually ends with minimum event cost histories given an event cost function. An inductive characterization of the solution space using different algebras for each granularity leads to efficient dynamic programming algorithms, ultimately ending with an O(mn) time complexity algorithm for computing the cost of a minimum-cost history (m and n: number of nodes in the input synteny and species trees). This time complexity matches that of the fastest known algorithms for classical gene reconciliation with transfers. We show how Synesth can be applied to infer Pareto-optimal evolutionary scenarios for CRISPR-Cas systems in a set of bacterial genomes.

Adaptive neural event‐triggered near‐optimal control for affined uncertain nonlinear discrete‐time system

AbstractA novel event‐triggered heuristic dynamic programming (HDP) algorithm is proposed for the near‐optimal control of uncertain discrete‐time nonlinear input‐affine systems. Based on input‐to‐state stability (ISS) analysis, a new event‐triggered mechanism (ETM) is designed. Under constant coefficients, a Lipschitz‐like assumption that forms the basis of the event‐triggering condition is considered to be conservative. To further reduce the conservativeness of the triggering condition and enlarge the average interevent time, an adaptive threshold parameter is utilized in the proposed ETM. In the HDP algorithm framework, model, critic, and action network are adopted to achieve state estimation, approximation to the optimal cost function, and solution to Hamilton–Jacobian–Bellman (HJB) equation. Under the proposed event‐triggered HDP algorithm, the closed system is proved to possess semiglobal uniform ultimate boundedness (SGUUB). Finally, by conducting simulation, it shows that on the premise of satisfying control performance, the event‐triggered strategy can realize reduction on the updating frequency of the controller.

Intelligent Low-Consumption Optimization Strategies: Economic Operation of Hydropower Stations Based on Improved LSTM and Random Forest Machine Learning Algorithm

The economic operation of hydropower stations has the potential to increase water use efficiency. However, there are some challenges, such as the fixed and unchangeable flow characteristic curve of the hydraulic turbines, and the large number of variables in optimal load distribution, which limit the progress of research. In this paper, we propose a new optimal method of the economic operation of hydropower stations based on improved Long Short-Term Memory neural network (I-LSTM) and Random Forest (RF) algorithm. Firstly, in order to accurately estimate the water consumption, the LSTM model’s hyperparameters are optimized using improved particle swarm optimization, and the I-LSTM method is proposed to fit the flow characteristic curve of the hydraulic turbines. Secondly, the Random Forest machine learning algorithm is introduced to establish a load-distribution model with its powerful feature extraction and learning ability. To improve the accuracy of the load-distribution model, we use the K-means algorithm to cluster the historical data and optimize the parameters of the Random Forest model. A Hydropower Station in China is selected for a case study. It is shown that (1) the I-LSTM method fits the operating characteristics under various working conditions and actual operating characteristics of hydraulic turbines, ensuring that they are closest to the actual operating state; (2) the I-LSTM method is compared with Support Vector Machine (SVM), Extreme Learning Machine (ELM) and Long Short-Term Memory neural network (LSTM). The prediction results of SVM have a large error, but compared with ELM and LSTM, MSE is reduced by about 46% and 38% respectively. MAE is reduced by about 25% and 21%, respectively. RMSE is reduced by about 27% and 24%, respectively; (3) the RF algorithm performs better than the traditional dynamic programming algorithm in load distribution. With the passage of time and the increase in training samples, the prediction accuracy of the Random Forest model has steadily improved, which helps to achieve optimal operation of the units, reducing their average total water consumption by 1.24%. This study provides strong support for the application of intelligent low-consumption optimization strategies in hydropower fields, which can bring higher economic benefits and resource savings to renewable energy production.

Route recombination for deterministic and non-deterministic orienteering problems with time windows: A dynamic programming approach

In this paper, we consider the Orienteering Problem with Time Windows (OPTW) where the goal is to visit customers within available time windows while maximizing the total reward provided by successive customer visits. For this problem, we introduce a new route recombination procedure that takes a set of solutions as input and returns the best combination containing at most k subsequences of customer visits from these solutions. This route recombination procedure is based on a dynamic programming algorithm enhanced with pruning strategies that can significantly reduce the size of the search space. It is also able to deal with time-dependent transition times. The experiments show that the algorithm proposed can be used as a lightweight and efficient post-optimization procedure working on elite solutions provided by a standard incomplete OPTW solver. Moreover, it can be used in a non-deterministic context where the reward values are not precisely known in advance; in this case, OPTW solutions can be first generated for various reward scenarios during an offline phase, and then combined during an online phase to quickly get a high-quality solution given the last-known reward values.

Research on Energy Management Strategy for Series Hybrid Tractor under Typical Operating Conditions Based on Dynamic Programming

In response to the issues of hybrid tractors’ energy management strategies, such as reliance on experience, difficulty in achieving optimal control, and incomplete analysis of typical operating conditions of tractors, an energy management strategy based on dynamic programming is proposed in combination with various typical operating conditions of tractors. This is aimed at providing a reference for the modeling and energy management strategies of series hybrid tractors. Taking the series hybrid tractor as the research object, the tractor dynamics models under three typical working conditions of plowing, rotary tillage, and transportation were established. With the minimum total fuel consumption of the tractor as the optimization target, the engine power as the control variable, and the state of charge of the power battery as the state variable, an energy management strategy based on a dynamic programming algorithm was established and simulation experiments were conducted. The simulation results show that, compared with the power-following energy management strategy, the energy management strategy based on the dynamic programming algorithm can reasonably control the operating state of the engine. Under the three typical working conditions of plowing, rotary tillage, and transportation, the battery SOC consumption increased by approximately 8.37%, 7.24%, and 0.77%, respectively, while the total fuel consumption decreased by approximately 25.28%, 21.54%, and 13.24%, respectively.