Dynamic Programming Research Articles

Optimization for fuel consumption and TCO of a heavy-duty truck with electricity-propelled trailer

As renewable energy grows in the heavy-duty truck sector, exploration of diverse solutions with varied energy types and powertrain configurations becomes a necessity. While most studies focus on powertrain configurations in tractors, this study takes a different approach: propose and evaluate new plug-in hybrid configurations with multiple power sources for heavy-duty trucks. The tractor retains its engine powertrain, while the trailer is equipped with electric axle(s), forming a fuel-electricity hybrid propulsion system. The specifications of the electric propulsion system are optimized, using a multi-objective particle swarm optimization algorithm and dynamic programming control algorithm to evaluate the energy consumption and total cost of ownership (TCO). The results highlight the potential of the configuration with three electric axles and single reduction gear in regard to vehicle energy consumption. Conversely, the configuration with one electric axle and a two-speed transmission exhibits the lowest TCO. Under the China World Transient Vehicle Cycle, the fuel consumption of the diesel truck is reduced by up to 44.59 % with the use of the optimized hybrid configuration. The truck's TCO can be then reduced by up to 832 thousand CNY in one-million-kilometer operation. For an actual road operation, these figures change to 34.79 % and 284 thousand CNY, respectively.

Learning‐based tracking control of AUV: Mixed policy improvement and game‐based disturbance rejection

AbstractA mixed adaptive dynamic programming (ADP) scheme based on zero‐sum game theory is developed to address optimal control problems of autonomous underwater vehicle (AUV) systems subject to disturbances and safe constraints. By combining prior dynamic knowledge and actual sampled data, the proposed approach effectively mitigates the defect caused by the inaccurate dynamic model and significantly improves the training speed of the ADP algorithm. Initially, the dataset is enriched with sufficient reference data collected based on a nominal model without considering modelling bias. Also, the control object interacts with the real environment and continuously gathers adequate sampled data in the dataset. To comprehensively leverage the advantages of model‐based and model‐free methods during training, an adaptive tuning factor is introduced based on the dataset that possesses model‐referenced information and conforms to the distribution of the real‐world environment, which balances the influence of model‐based control law and data‐driven policy gradient on the direction of policy improvement. As a result, the proposed approach accelerates the learning speed compared to data‐driven methods, concurrently also enhancing the tracking performance in comparison to model‐based control methods. Moreover, the optimal control problem under disturbances is formulated as a zero‐sum game, and the actor‐critic‐disturbance framework is introduced to approximate the optimal control input, cost function, and disturbance policy, respectively. Furthermore, the convergence property of the proposed algorithm based on the value iteration method is analysed. Finally, an example of AUV path following based on the improved line‐of‐sight guidance is presented to demonstrate the effectiveness of the proposed method.

Discrete‐Time Optimal Control of State‐Constrained Nonlinear Systems Using Approximate Dynamic Programming

ABSTRACTThis article investigates the optimal control problem (OCP) for a class of discrete‐time nonlinear systems with state constraints. First, to overcome the challenge caused by the constraints, the original constrained OCP is transformed into an unconstrained OCP by utilizing the system transformation technique. Second, a new cost function is designed to alleviate the effect of system transformation on the optimality of the original system. Further, a novel off‐policy deterministic approximate dynamic programming (ADP) scheme is developed to obtain a near‐optimal solution for the transformed OCP. Compared to existing off‐policy deterministic ADP schemes, the developed scheme relaxes the requirement on the learning data and saves computing resources from the perspective of training neural networks. Third, considering approximation errors, we analyze the convergence and stability of the developed ADP scheme. Finally, the developed ADP with the designed cost function is tested in two numerical cases, and simulation results confirm its effectiveness.

Dynamic Programming-Based Approach to Model Antigen-Driven Immune Repertoire Synthesis

This paper presents a novel approach to modeling the repertoire of the immune system and its adaptation in response to the evolutionary dynamics of pathogens associated with their genetic variability. It is based on application of a dynamic programming-based framework to model the antigen-driven immune repertoire synthesis. The processes of formation of new receptor specificity of lymphocytes (the growth of their affinity during maturation) are described by an ordinary differential equation (ODE) with a piecewise-constant right-hand side. Optimal control synthesis is based on the solution of the Hamilton–Jacobi–Bellman equation implementing the dynamic programming approach for controlling Gaussian random processes generated by a stochastic differential equation (SDE) with the noise in the form of the Wiener process. The proposed description of the clonal repertoire of the immune system allows us to introduce an integral characteristic of the immune repertoire completeness or the integrative fitness of the whole immune system. The quantitative index for characterizing the immune system fitness is analytically derived using the Feynman–Kac–Kolmogorov equation.

Multi-stage planning of clean resources and energy storage assets with hybrid uncertainty modeling for low-carbon resilient distribution systems

Climate change drives the urgent need for low-carbon and resilient energy system transitions. However, current planning methods ignore the inherent conflicts between carbon emission reduction and resilience enhancement, failing to optimally balance asset allocation for both aspects. They also neglect the long-term dynamic and stochastic nature of transitions, exposing systems to hybrid uncertainties. This paper presents a multi-stage dynamic planning method for clean resources and energy storage assets in power distribution networks. First, to facilitate low-carbon and resilient transitions, adaptive, stage-wise planning decisions are optimally determined under various planning strategies to mitigate risks stemming from hybrid uncertainties. Second, to precisely quantify the impact of hybrid uncertainties on decision-making, a systematic hybrid-uncertainty modeling approach is designed to capture short- and long-term uncertainties arising from both exogenous and endogenous factors. Third, to alleviate computational complexity, a decomposition-based stochastic dynamic dual integer programming method with augmented cut generation is introduced, which decomposes the original problem into stage-wise subproblems coordinated through forward and backward steps. Numerical results show that the proposed method generates cost-effective planning schemes, yielding $60 K in savings and reducing carbon emissions by 3853 tons. The method also demonstrates strong scalability, with a 52% reduction in solution time for large-scale planning problems.

Stochastic measurement-based multi-cloud consolidation for efficient resource distribution

In distributed cloud systems, due to high computational complexity, existing methods cannot infinitely reduce the measurement interval to obtain the mean demand. However, considering both the unpredictability of demand and the diversity of pricing adds significant complexity to the problem. In response, we introduce an efficient algorithm called Stochastic Demand-oriented Resource consolidation based on Meta-heuristics (SDRM). The algorithm introduces the stochastic demand model and formulates a cross-cloud demand consolidation and resource scheduling problem, and tries to solve it efficiently by combining differential evolution algorithm with dynamic programming optimization. The results are provided for experiments utilizing both simulated and real-world data. It demonstrates that compared with traditional algorithms, increasing gain by up to 53%, SDRM just incurs an affordable time expense (i.e., 1.10-2.51 times that of existing algorithms). Therefore, SDRM stands as a compelling solution for resource distribution in heterogeneous cloud environments.

Pension funds with longevity risk: an optimal portfolio insurance approach

We present a unified framework designed to provide an optimal investment strategy for members of a defined contribution pension plan. Our model guarantees a minimum retirement savings level, expressed as a target annuity, by assuming uncertainty in interest rates, labor income, and mortality during the accumulation phase. To protect the accumulated retirement capital against both investment and longevity risks, the present value of the guaranteed lifetime annuity is regarded as the baseline wealth to hold upon reaching the retirement date, while a purpose-oriented proportion portfolio strategy is employed to invest the residual wealth. By applying standard dynamic programming techniques, we determine a closed-form solution to the stochastic control problem with the objective of maximizing the expected utility of the final surplus, defined as the difference between the accumulated wealth and the target annuity value. The theoretical findings are bolstered by a comprehensive numerical analysis designed to assess the impact of longevity on investment policies, highlighting the suitability of our proposal for managing defined contribution schemes.

Acoustography by Beam Engineering and Acoustic Control Node: BEACON.

Acoustic manipulation has emerged as a valuable tool for precision controls and dynamic programming of cells and particles. However, conventional acoustic manipulation approaches lack the finesse necessary to form intricate, configurable, continuous, and 3D patterning of particles. Here, this study reports acoustography by Beam Engineering and Acoustic Control Node (BEACON), which delivers intricate, configurable patterns by guiding particles along custom paths with independent phase modulation. Leveraging analytical methods of orbital angular momentum beam via iterative Wirtinger hologram algorithm, this study accomplish acoustography by facilitating orbital angular momentum traps, enabling continuous 2D and 3D acoustic manipulation of microparticles in any desired geometry, with phase modulation independent of intensity. Utilizing on-chip acoustography, the BEACON platform markedly increases the space-bandwidth product to 31000 while attaining an enhanced resolution with a pixel size of ≈25µm, surpassing the typical resolution of over 200µm in previous holographic particle manipulation methods. The capabilities of BEACON are demonstrated in creating intricate triple helical tracing structures using microdroplets (20µm in diameter) and those carrying DNA to validate the effectiveness of the acoustography and phase control methods. This study offers new particle manipulation opportunities, paving the way for next-generation biomedical systems and the future of contact-free precision manufacturing.

Branch-and-Bound and Dynamic Programming Approaches for the Knapsack Problem

Branch-and-Bound and Dynamic Programming Approaches for the Knapsack Problem

Resilient consensus of heterogeneous multi-agent systems under asynchronous optimal DoS attack schedules

This study aims to address the challenge of achieving resilient consensus in heterogeneous multi-agent systems (MASs) under asynchronous optimal Denial of Service (DoS) attack schedules. Considering the start and duration of adversaries vary from different communication channels of MASs, we mainly focus on how adversaries with limited energy choose channels to launch attacks to cause the greatest damage to system performance. Based on dynamic programming, an asynchronous optimal DoS attack schedules generation algorithm is obtained. Optimal DoS attack schedules provides a basis for designing more effective secure control methods. Then, a distributed resilient control strategy that utilizes an observer-based approach to achieve output consensus in heterogeneous MASs under asynchronous optimal DoS attack schedules is proposed. Additionally, we establish conditions regarding the durations and frequencies of DoS attacks as well. Finally, two simulations are performed to showcase the efficacy of our proposed method.

MPPT control for PMSG based wind turbine: Adaptive dynamic programming approach

This paper proposes an online adaptive dynamic programming-based controller for the variable-speed wind turbine system, ensuring it tracks the maximum power point under uncertain conditions. This control scheme is composed of two components: the adaptive optimal control component and the high-order disturbance observer-based adaptive sliding mode control component. The adaptive optimal control part is designed using the online adaptive dynamic programming technique, which is one of the algorithms of the reinforcement learning method. This control component has the role of achieving the optimal characteristics for the nonlinear system. Likewise, the high-order disturbance observer is established to estimate system uncertainties and external disturbances. These approximated disturbances are then fed to the adaptive sliding mode control component for system disturbance compensation. As a result, the system, which is affected by the unknown disturbances, achieves robust optimal performance. In addition, the convergence of the adaptive laws and the stability of the overall system are ensured through the Lyapunov theory. Finally, comparative simulations are conducted to validate the advantages of the proposed control scheme in comparison to the existing methods.

Anti-factor is FPT Parameterized by Treewidth and List Size (but Counting is Hard)

AbstractIn the general AntiFactor problem, a graph G and, for every vertex v of G, a set $$X_v\subseteq {\mathbb {N}}$$ X v ⊆ N of forbidden degrees is given. The task is to find a set S of edges such that the degree of v in S is not in the set $$X_v$$ X v . Standard techniques (dynamic programming plus fast convolution) can be used to show that if M is the largest forbidden degree, then the problem can be solved in time $$(M+2)^{{{\text {tw}}}}\cdot n^{{\mathcal {O}}(1)}$$ ( M + 2 ) tw · n O ( 1 ) if a tree decomposition of width $${{\text {tw}}}$$ tw is given. However, significantly faster algorithms are possible if the sets $$X_v$$ X v are sparse: our main algorithmic result shows that if every vertex has at most $$x$$ x forbidden degrees (we call this special case AntiFactorx), then the problem can be solved in time $$(x+1)^{{\mathcal {O}}({{\text {tw}}})}\cdot n^{{\mathcal {O}}(1)}$$ ( x + 1 ) O ( tw ) · n O ( 1 ) . That is, AntiFactorx is fixed-parameter tractable parameterized by treewidth $${{\text {tw}}}$$ tw and the maximum number $$x$$ x of excluded degrees. Our algorithm uses the technique of representative sets, which can be generalized to the optimization version, but (as expected) not to the counting version of the problem. In fact, we show that #AntiFactor1 is already #$$[1]$$ [ 1 ] -hard parameterized by the width of the given decomposition. Moreover, we show that, unlike for the decision version, the standard dynamic programming algorithm is essentially optimal for the counting version. Formally, for a fixed nonempty set $$X$$ X , we denote by $$X$$ X -AntiFactor the special case where every vertex v has the same set $$X_v=X$$ X v = X of forbidden degrees. We show the following lower bound for every fixed set $$X$$ X : if there is an $$\epsilon >0$$ ϵ > 0 such that #$$X$$ X -AntiFactor can be solved in time $$(\max X+2-\epsilon )^{{{\text {tw}}}}\cdot n^{{\mathcal {O}}(1)}$$ ( max X + 2 - ϵ ) tw · n O ( 1 ) given a tree decomposition of width $${{\text {tw}}}$$ tw , then the counting strong exponential-time hypothesis (#SETH) fails.

A fixed point theorem in T-partial b-metric spaces and an application in dynamic programming

Here, a fixed point theorem is proved in a new class of spaces called [Formula: see text]-partial [Formula: see text]-metric spaces without using the compactness. This class of spaces is a new generalization of [Formula: see text]-metric spaces such that the generated topology is a Hausdorff topology. Moreover, an application in dynamic programming is given to illustrate the usability of the obtained results.

Mixed‐Integer Optimal Control via Reinforcement Learning: A Case Study on Hybrid Electric Vehicle Energy Management

ABSTRACTMany optimal control problems require the simultaneous output of discrete and continuous control variables. These problems are typically formulated as mixed‐integer optimal control (MIOC) problems, which are challenging to solve due to the complexity of the solution space. Numerical methods such as branch‐and‐bound are computationally expensive and undesirable for real‐time control. This article proposes a novel hybrid‐action reinforcement learning (HARL) algorithm, twin delayed deep deterministic actor‐Q (TD3AQ), for MIOC problems. TD3AQ leverages actor‐critic and Q‐learning methods to manage discrete and continuous action spaces simultaneously. The proposed algorithm is evaluated on a plug‐in hybrid electric vehicle (PHEV) energy management problem, where real‐time control of the discrete variables, clutch engagement/disengagement and gear shift, and continuous variable, engine torque, is essential to maximize fuel economy while satisfying driving constraints. Simulation results show that TD3AQ achieves near‐optimal control, with only a 4.69% difference from dynamic programming (DP), and outperforms baseline reinforcement learning algorithms for hybrid action spaces. The sub‐millisecond execution time indicates potential applicability in other time‐critical scenarios, such as autonomous driving or robotic control.

Packing squares independently

Given a set of squares and a strip with bounded width and infinite height, we consider a square strip packaging problem, which we call the square independent packing problem (SIPP), to minimize the strip height so that all the squares are packed into independent cells separated by horizontal and vertical partitions. For the SIPP, we first investigate efficient solution representations and propose a compact representation that reduces the search space from Ω(n!) to O(2n), with n the number of given squares, while guaranteeing that there exists a solution representation that corresponds to an optimal solution. Based on the solution representation, we show that the problem is NP-hard. To solve the SIPP, we propose a dynamic programming method that can be extended to a fully polynomial-time approximation scheme (FPTAS). We also propose three mathematical programming formulations based on different solution representations and confirm their performance through computational experiments with a mathematical programming solver. Finally, we discuss several extensions that are relevant to practical applications.

Stochastic representation for solutions of a system of coupled HJB-Isaacs equations with integral–differential operators

In this paper, we focus on the stochastic representation of a system of coupled Hamilton–Jacobi–Bellman–Isaacs (HJB–Isaacs (HJBI), for short) equations which is in fact a system of coupled Isaacs’ type integral-partial differential equation. For this, we introduce an associated zero-sum stochastic differential game, where the state process is described by a classical stochastic differential equation (SDE, for short) with jumps, and the cost functional of recursive type is defined by a new type of backward stochastic differential equation (BSDE, for short) with two Poisson random measures, whose wellposedness and a prior estimate as well as the comparison theorem are investigated for the first time. One of the Poisson random measures μ appearing in the SDE and the BSDE stems from the integral term of the HJBI equations; the other random measure in BSDE is introduced to link the coupling factor of the HJBI equations. We show through an extension of the dynamic programming principle that the lower value function of this game problem is the viscosity solution of the system of our coupled HJBI equations. The uniqueness of the viscosity solution is also obtained in a space of continuous functions satisfying certain growth condition. In addition, also the upper value function of the game is shown to be the solution of the associated system of coupled Isaacs’ type of integral-partial differential equations. As a byproduct, we obtain the existence of the value for the game problem under the well-known Isaacs’ condition.

A holistic matrix norm-based alternative solution method for Markov reward games

In this study, we focus on examining single-agent stochastic games, especially Markov reward games represented in the form of a decision tree. We propose an alternative solution method based on the matrix norms for these games. In contrast to the existing methods such as value iteration, policy iteration, and dynamic programming, which are state-and-action-based approaches, the proposed matrix norm-based method considers the relevant stages and their actions as a whole and solves it holistically for each stage without computing the effects of each action on each state's reward individually. The new method involves a distinct transformation of the decision tree into a payoff matrix for each stage and the utilization of the matrix norm of the obtained payoff matrix. Additionally, the concept of the moving matrix is integrated into the proposed method to incorporate the impacts of all actions on the stage simultaneously, rendering the method holistic. Moreover, we present an explanatory algorithm for the implementation of the method and also provide a comprehensive solution diagram explaining the method figuratively. As a result, we offer a new and alternative perspective for solving the games with the help of the proposed method due to the simplicity of utilization of the matrix norms in addition to the existing methods. For clarification of the matrix norm-based method, we demonstrate the figurative application of the method on a benchmark Markov reward game with 2-stages and 2-actions and a comprehensive implementation of the method on a game consisting of 3-stages and 3-actions.

ADP-based adaptive control of ship course tracking system with prescribed performance

In this paper, an adaptive dynamic programing (ADP)-based adaptive control problem is studied for the ship course tracking system with prescribed performance. Firstly, the complex sea conditions will bring difficulty to ship course tracking when unmanned ship sailing on sea. In order to overcome this difficulty, a prescribed performance feedforward term is designed in the controller to improve the course tracking ability of ship. Secondly, by defining a cost function, an ADP technology is exploited to achieve an optimal control of ship course tracking system. Finally, the stability of the closed loop system is turned out by Lyapunov stability theory and all signals are uniformly ultimately bounded (UUB). Simulation results validate the effectiveness of the proposed method.

Automated pricing and replenishment forecasting for vegetable items

Firstly, the thesis extensively analysed and processed the vegetable sales data of a fresh food superstore. It integrated the product information and sales data of each vegetable category, and carried out data cleaning and analysis through Excel tools, including sorting out the outliers, listing the sales volume of each of the six categories, using pivot tables for statistical analysis, and establishing line graphs, heat maps, scatter plots, and other visual displays of the data characteristics and correlations. Secondly, the thesis investigated the relationship between sales volume and pricing, fitted the correlation between total sales volume and pricing using a random forest regression model, and predicted the daily replenishment volume for the coming week using an LSTM time series model. It also fitted quantitative fitting formulas for metrics for each vegetable category, classified profits into normal sales volume profits and discounted sales volume profits, and ultimately maximised profits through dynamic programming models and particle swarm optimisation algorithms. Finally, the thesis considered the impact of other factors such as weather, holidays, seasons and market environment on sales volume with specific data collection and analyses that help guide better pricing strategies. The findings will contribute to a better understanding of vegetable sales behaviour and optimise superstore operational strategies.

MDSA: A Dynamic and Greedy Approach to Solve the Minimum Dominating Set Problem

The graph theory is one of the fundamental structures in computer science used to model various scientific and engineering problems. Many problems within the graph theory are categorized as NP-hard and NP-complete. One such problem is the minimum dominating set (MDS) problem, which seeks to identify the minimum possible subsets in a graph such that every other node in the subset is directly connected to a node in this subset. Due to its inherent complexity, developing an efficient polynomial-time method to address the MDS problem remains a significant challenge in graph theory. This paper introduces a novel algorithm that utilizes a centrality measure known as the Malatya Centrality to effectively address the MDS problem. The proposed algorithm, called the Malatya Dominating Set Algorithm (MDSA), leverages centrality values to identify dominating sets within a graph. It extends the Malatya centrality by incorporating a second-level centrality measure, which enhances the identification of dominating nodes. Through a systematic and algorithmic approach, these centrality values are employed to pinpoint the elements of the dominating set. The MDSA uniquely integrates greedy and dynamic programming strategies. At each step, the algorithm selects the most optimal (or near-optimal) node based on the centrality values (greedy approach) while updating the neighboring nodes’ criteria to influence subsequent decisions (dynamic programming). The proposed algorithm demonstrates efficient performance, particularly in large-scale graphs, with time and space requirements scaling proportionally with the size of the graph and its average degree. Experimental results indicate that our algorithm outperforms existing methods, especially in terms of time complexity when applied to large datasets, showcasing its effectiveness in addressing the MDS problem.