Dynamic Programming Framework Research Articles

Adaptive Dynamic Programming for Robust Event-Driven Tracking Control of Nonlinear Systems With Asymmetric Input Constraints.

This article considers the robust dynamic event-driven tracking control problem of nonlinear systems having mismatched disturbances and asymmetric input constraints. Initially, to tackle the asymmetric constraints, a novel nonquadratic value function is constructed for the original system. This makes the asymmetrically constrained tracking control problem transformed into an unconstrained optimal regulation problem. Then, a dynamic event-driven mechanism is proposed. Meanwhile, the event-driven Hamilton-Jacobi-Bellman equation (ED-HJBE) is developed for the optimal regulation problem in order to acquire the optimal control with distinctly decreased computational burden. To solve the ED-HJBE, a single critic neural network (CNN) is designed in the adaptive dynamic programming framework. Meanwhile, the gradient descent method is employed to update the CNN's weights. After that, both the weight estimation error and the tracking error are proved to be uniformly ultimately bounded via Lyapunov's direct method. Finally, simulations of the spring-mass-damper system and the pendulum plant are separately utilized to validate the established theoretical claims.

Dynamic event-based cooperative control of virtually coupled high-speed trains: an ADP-based optimal control scheme

This paper is concerned with the cooperative control issue of virtually coupled high-speed trains (VCHSTs) under an adaptive dynamic programming (ADP) framework, where a dynamic event-triggered mechanism is employed to govern the communication between sensors and observers. A learning-based observer, equipped with a robust term, is exploited to mitigate the effect of the unknown nonlinear dynamics of the VCHSTs. Furthermore, to realise the distributed design, an auxiliary control signal is employed to simply the closed-loop system and also integrated into the cost function to handle the coupled nature of the investigated systems. The Hamilton–Jacobi–Bellman (HJB) equation is approximated by actor and critic networks in the ADP framework, which iteratively acquires the desired control strategy. By resorting to the common Lyapunov stability, some sufficient conditions are derived to achieve the desired observer gain and learning rates of NNs. Ultimately, the control scheme's efficiency is confirmed by simulations of VCHSTs.

Efficient Seeding for Error-Prone Sequences with SubseqHash2.

Seeding is an essential preparatory step for large-scale sequence comparisons. Substring-based seeding methods such as kmers are ideal for sequences with low error rates but struggle to achieve high sensitivity while maintaining a reasonable precision for error-prone long reads. SubseqHash, a novel subsequence-based seeding method we recently developed, achieves superior accuracy to substring-based methods in seeding sequences with high mutation/error rates, while the only drawback is its computation speed. In this paper, we propose SubseqHash2, an improved algorithm that can compute multiple sets of seeds in one run by defining orders over all length- subsequences and identifying the optimal subsequence under each of the orders in a single dynamic programming framework. The algorithm is further accelerated using SIMD instructions. SubseqHash2 achieves a 10-50× speedup over repeating SubseqHash while maintaining the high accuracy of seeds. We demonstrate that SubseqHash2 drastically outperforms popular substring-based methods including kmers, minimizers, syncmers, and Strobemers for three fundamental applications. In read mapping, SubseqHash2 can generate adequate seed-matches for aligning hard reads that minimap2 fails on. In sequence alignment, SubseqHash2 achieves high coverage of correct seeds and low coverage of incorrect seeds. In overlap detection, seeds produced by SubseqHash2 lead to more correct overlapping pairs at the same false-positive rate. With all the algorithmic breakthroughs of SubseqHash2, we clear the path for the wide adoption of subsequence-based seeds in long-read analysis. SubseqHash2 is available at https://github.com/Shao-Group/SubseqHash2.

Event-Triggered Constrained Optimal Control for Organic Rankine Cycle Systems via Safe Reinforcement Learning.

The organic Rankine cycle (ORC) is an effective application for converting low-grade heat sources into power and is crucial for environmentally friendly production and energy recovery. However, the inherent complexity of the mechanism, its strong and unidentified nonlinearity, and the presence of control constraints severely impair the design of its optimal controller. To solve these issues, this study provides a novel event-triggered (ET) constrained optimal control approach for the ORC systems based on a safe reinforcement learning technique to find the optimal control law. Instead of employing the usual non-quadratic integral form to solve the control-limited optimal control problems, a constraint handling strategy based on a relaxed weighted barrier function (BF) technique is proposed. By adding the BF terms to the original value function, a modified value iteration algorithm is developed to make the control input solutions that tend to violate the constraints be pushed back and maintained in their safe sets. In addition, the ET mechanism proposed in this article is critically required for the ORC systems, and it can significantly reduce the computational load. The combination of these two techniques allows the ORC systems to achieve set-point tracking control and satisfy the control restrictions. The proposed approach is conducted based on a heuristic dynamic programming framework with three neural networks (NNs) involved. The safety and convergence of the proposed approach and the stability of the closed-loop system are analyzed. Simulation results and comparisons are presented to demonstrate its effectiveness.

Novel Discounted Adaptive Critic Control Designs With Accelerated Learning Formulation.

Inspired by the successive relaxation method, a novel discounted iterative adaptive dynamic programming framework is developed, in which the iterative value function sequence possesses an adjustable convergence rate. The different convergence properties of the value function sequence and the stability of the closed-loop systems under the new discounted value iteration (VI) are investigated. Based on the properties of the given VI scheme, an accelerated learning algorithm with convergence guarantee is presented. Moreover, the implementations of the new VI scheme and its accelerated learning design are elaborated, which involve value function approximation and policy improvement. A nonlinear fourth-order ball-and-beam balancing plant is used to verify the performance of the developed approaches. Compared with the traditional VI, the present discounted iterative adaptive critic designs greatly accelerate the convergence rate of the value function and reduce the computational cost simultaneously.

Economic optimization of Wolbachia-infected Aedes aegypti release to prevent dengue.

Dengue virus, primarily transmitted by the Aedes aegypti mosquito, is a major public health concern affecting ≈3.83 billion people worldwide. Recent releases of Wolbachia-transinfected Ae. aegypti in several cities worldwide have shown that it can reduce dengue transmission. However, these releases are costly, and, to date, no framework has been proposed for determining economically optimal release strategies that account for both costs associated with disease risk and releases. We present a flexible stochastic dynamic programming framework for determining optimal release schedules for Wolbachia-transinfected mosquitoes that balances the cost of dengue infection with the costs of rearing and releasing transinfected mosquitoes. Using an ordinary differential equation model of Wolbachia and dengue in a hypothetical city loosely describing areas at risk of new dengue epidemics, we determined that an all-or-nothing release strategy that quickly brings Wolbachia to fixation is often the optimal solution. Based on this, we examined the optimal facility size, finding that it was inelastic with respect to the mosquito population size, with a 100% increase in population size resulting in a 50-67% increase in optimal facility size. Furthermore, we found that these results are robust to mosquito life-history parameters and are mostly determined by the mosquito population size and the fitness costs associated with Wolbachia. These results reinforce that Wolbachia-transinfected mosquitoes can reduce the cost of dengue epidemics. Furthermore, they emphasize the importance of determining the size of the target population and fitness costs associated with Wolbachia before releases occur. © 2024 The Authors. Pest Management Science published by John Wiley & Sons Ltd on behalf of Society of Chemical Industry.

SeaTE: Subjective ex ante Treatment Effect of Health on Retirement

The paper studies the effect of health on work among older workers by eliciting two- and four-year-ahead subjective probabilities of working under alternative health states. These measures predict work outcomes. Person-specific differences in working probabilities across health states can be interpreted as Subjective ex ante Treatment Effects (SeaTEs) in a potential outcomes framework and as taste for work within a discrete choice dynamic programming framework. There is substantial heterogeneity in expectations of work conditional on health. The paper shows how heterogeneity in taste for work correlated with health can bias regression estimates of the effect of health on retirement. (JEL D84, I12, J14, J22, J26)

Design and analysis of self-priming extension DNA hairpin probe for miRNA detection based on a unified dynamic programming framework

MicroRNAs (miRNAs) are potential biomarkers for cancer diagnosis and prognosis, methods for detecting miRNAs with high sensitivity, selectivity, and stability are urgently needed. Various nucleic acid probes that have traditionally been for this purpose suffer several drawbacks, including inefficient signal-to-noise ratios and intensities, high cost, and time-consuming method establishment. Computing tools used for investigating the thermodynamics of DNA hybridization reactions can accurately predict the secondary structure of DNA and the interactions between DNA molecules. Herein, NUPACK was used to design a series of nucleic acid probes and develop a phosphorothioated-terminal hairpin formation and self-priming extension (PS-THSP) signal amplification strategy, which enabled the ultrasensitive detection of miR-200a in serum samples. The free and binding energies of the DNA detection probes calculated using NUPACK, as well as the biological experimental results, were considered synthetically to select the best sequence and experimental conditions. A unified dynamic programming framework, NUPACK analysis and the experimental data, were complementary and improved the designed model in all respects. Our study demonstrates the feasibility of using computer technology such as NUPACK to simplify the experimental process and provide intuitive results.

Energy-saving time allocation strategy with uncertain dwell times in urban rail transit: Two-stage stochastic model and nested dynamic programming framework

During practical operations, the urban rail transit system suffers from various uncertainties, particularly uncertain dwell times, which significantly impact the execution of the timetable and affect its performance, regarding train energy consumption and timetable stability. Using multi-scenario dwell times to capture its uncertainty, in this study, a two-stage chance-constrained stochastic model involving a section-time allocation stage and an optimal driving strategy stage is developed to minimize the expected energy consumption and improve stability. An exact nested dynamic programming (NDP) framework was designed to solve the model. The effectiveness and performance of the proposed methodology were investigated using a series of numerical experiments based on a small-scale instance from the Beijing Yizhuang Line. The optimized section-time allocation strategy reduced the expected energy consumption by 6.6% and improved the the minimum stability ratio by 8.5% by selecting the appropriate weight ratio. Four other sensitivity analyses were conducted to provide realistic managerial insights. Finally, a large-scale study of the Beijing 4-Daxing Subway Line was conducted to validate the scalability and efficiency of the NDP framework.

Supplementary Control for Quantized Discrete-Time Nonlinear Systems Under Goal Representation Heuristic Dynamic Programming.

This article is concerned with supplementary control of discrete-time nonlinear systems with multiple controllers in the framework of goal representation heuristic dynamic programming (GrHDP), where a logarithmic quantizer is used to govern the network communication. For the addressed problem, a neural network (NN)-based observer is first proposed to estimate the unknown system state in the simultaneous presence of quantized influence. In light of the estimated states and the ideal control inputs via a zero-sum game, a GrHDP algorithm with a reinforced term is developed to implement the supplementary control task, where some novel weight updating rules are constructed by virtue of an additional tunable parameter to improve the system performance. Furthermore, a set of conditions about the stability of estimated error dynamics of both observer states and updated NNs' weights are derived by resorting to the Lyapunov stability theory. Finally, the effectiveness of the developed method is verified by a power system and a numerical experiment.

ADP-Based Event-Triggered Constrained Optimal Control on Spatiotemporal Process: Application to Temperature Field in Roller Kiln.

The precise control of the spatiotemporal process in a roller kiln is crucial in the production of Ni-Co-Mn layered cathode material of lithium-ion batteries. Since the product is extremely sensitive to temperature distribution, temperature field control is of great significance. In this article, an event-triggered optimal control (ETOC) method with input constraints for the temperature field is proposed, which takes up an important position in reducing the communication and computation costs. A nonquadratic cost function is adopted to describe the system performance with input constraints. First, we present the problem description of the temperature field event-triggered control, where this field is described by a partial differential equation (PDE). Then, the event-triggered condition is designed according to the information of system states and control inputs. On this basis, a framework of the event-triggered adaptive dynamic programming (ETADP) method that is based on the model reduction technology is proposed for the PDE system. A critic network is used to approach the optimal performance index by a neural network (NN) together with that an actor network is used to optimize the control strategy. Furthermore, an upper bound of the performance index and a lower bound of interexecution times, as well as the stabilities of the impulsive dynamic system and the closed-loop PDE system, are also proved. Simulation verification demonstrates the effectiveness of the proposed method.

A dynamic programming approach for energy management in hybrid electric vehicles under uncertain driving conditions

ABSTRACT This paper addresses the limited adaptability and the computational burden of energy management systems (EMSs) for hybrid electric vehicles (HEVs) implemented via dynamic programming (DP)-based approaches. First, a deterministic dynamic programming (DDP) framework is presented to solve HEV EMS problems subject to a specific driving cycle. To address this limitation, an improved DDP approach, integrating the actual travelled position of the vehicle into the control law, is proposed. This way, a given DDP-based EMS can be applied to all the driving cycles, yet still measured on the same road. Stochastic dynamic programming (SDP)-based EMSs are also developed and prove to be more adaptive to driving scenarios completely different from the ones used for their computation. Real-world driving cycles are employed in all the presented cases, while a reduced HEV powertrain model is used to alleviate the typical DP computational burden.

Multilayer Iterative Stochastic Dynamic Programming for Optimal Energy Management of Residential Loads with Electric Vehicles

This work introduces a multilayer iterative stochastic dynamic programming (MISDP) framework for optimizing energy management in smart residential settings, incorporating electric vehicles to reduce energy costs while enhancing operational efficiency. The study investigates the complexities of managing residential loads with integrated EV batteries, set against the backdrop of unpredictable charging demands and fluctuating energy prices. The proposed method is designed to optimize charging and discharging schedules, ensuring cost‐effective energy consumption without compromising the longevity of EV’s battery operations. The proposed MISDP strategy encompasses multi‐iteration processes, both at internal and external levels, that not only highlight the method’s capacity for precise, real‐time decision‐making but also underscore its adaptability to the dynamic nature of energy systems. The external iteration primarily focuses on adapting to broader operational variables, such as fluctuating prices and demand patterns, setting a framework for optimization. Concurrently, the internal iteration updates the details of EV battery operation, fine‐tuning charging and discharging strategies to refine the control law sequence for each operational period, ensuring optimal energy management. Throughout the iteration process, the framework ensures the performance index function remains bounded, adhering strictly to the evolving control law sequence. Through comparative analysis, the MISDP framework is evaluated against different optimization techniques, demonstrating its superior capability in achieving significant energy cost savings and operational effectiveness while ensuring convergence under stochastic conditions.

A Dynamic Programming Framework for Optimal Planning of Redundant Robots Along Prescribed Paths With Kineto-Dynamic Constraints

A Dynamic Programming Framework for Optimal Planning of Redundant Robots Along Prescribed Paths With Kineto-Dynamic Constraints

Partially observed optimal control of local and remote controllers

AbstractThe article investigates a networked optimal control problem with one remote controller and local controllers, where each local controller partially observes the local state and sends the observed information to remote controller through an unreliable channel and yet the channels from remote controller to local controllers are perfect. The objective of all the controllers is to cooperate to minimize a common cost function. In order to overcome the difficulties caused by partial observation of state, sufficient statistic of historical data is firstly investigated for a nonlinear non‐Gaussian partial observation model, which takes the same role as that of state in the full observation model. Then, a dynamic programming framework is given for solving the optimal control problem based on the sufficient statistic and common information approach. Furthermore, the explicit closed‐form solution is obtained for a linear‐quadratic‐Gaussian optimal control problem. Finally, a numerical example is given to show the effectiveness of the obtained results of this article.

Model-free adaptive dynamic event-triggered robust control for unknown nonlinear systems using iterative neural dynamic programming

This paper proposes an adaptive dynamic event-triggered (ADET) robust control method for unknown nonlinear systems using iterative neural dynamic programming (INDP). Firstly, the ADET robust control problem is transformed into an ADET optimal control problem by introducing an infinite domain integral function. Then, dynamic variables and adaptive thresholds are designed for discrete-time systems with auxiliary variables. The proposed ADET method reduces computation and transmission costs compared to existing static and dynamic event triggering methods. INDP is utilized to learn the optimal solution of the ADET Hamilton-Jacobi-Bellman equation within the heuristic dynamic programming (HDP) framework, providing system stability and algorithm convergence. The INDP algorithm employs model, action, and critic neural networks and includes a method to directly minimize the iterative cost function in the back-propagation process. The neural network implementation of the INDP algorithm is detailed. Simulation results demonstrate that the proposed ADET method based on INDP achieves faster convergence with fewer transmitted data and control updates.

Learning‐based collision avoidance and robust H∞ optimal formation control for uncertain quadrotor UAV systems

AbstractThis paper investigates the robust optimal formation control for the position subsystem of quadrotor unmanned aerial vehicles (UAVs) subject to external disturbances and collision constraints. To prevent collision with both members of the formation and external obstacles, a collision avoidance potential function is constructed using relative position and velocity information. The basic bounded control input can ensure the stable flight and collision avoidance of a quadrotor UAV system. Based on the approximate dynamic programming (ADP) framework and two‐player zero‐sum differential game theory, the optimal controller is designed to further enhance the control performance of the system. The optimal value function is approximated by a single layer neural network, which avoids solving complex nonlinear Hamilton‐JacobiIsaac (HJI) equation. The stability of the closed loop system is proved. The effectiveness of the robust optimal formation controller based on learning is validated through MATLAB simulation in two distinct scenarios.

Display optimization under the multinomial logit choice model: Balancing revenue and customer satisfaction

In this paper, we consider an assortment optimization problem in which a platform must choose pairwise disjoint sets of assortments to offer across a series of T stages. Arriving customers begin their search process in the first stage, and progress sequentially through the stages until their patience expires, at which point they make a multinomial logit–based purchasing decision from among all products they have viewed throughout their search process. The goal is to choose the sequential displays of product offerings to maximize expected revenue. Additionally, we impose stage‐specific constraints that ensure that as each customer progresses farther and farther through the T stages, there is a minimum level of “desirability” met by the collections of displayed products. We consider two related measures of desirability: purchase likelihood and expected utility derived from the offered assortments. In this way, the offered sequence of assortments must be both high earning and well liked, which breaks from the traditional assortment setting, where customer‐centric considerations are generally not explicitly accounted for. We show that our assortment problem of interest is strongly NP‐Hard, thus ruling out the existence of a fully polynomial‐time approximation scheme (FPTAS). From an algorithmic standpoint, as a warm‐up, we develop a simple constant factor approximation scheme in which we carefully stitch together myopically selected assortments for each stage. Our main algorithmic result consists of a polynomial‐time approximation scheme (PTAS), which combines a handful of structural results related to the make‐up of the optimal assortment sequence within an approximate dynamic programming framework. We also provide an additional approximation scheme, which, under mild assumptions, can handle a cardinality constraint that enforces that an exact number of new products are introduced at each stage. Using an extensive set of numerical experiments, we demonstrate that both algorithms exhibit excellent practical performance, producing sequences of assortments that are, on average, always within 2% of optimal.

Optimal battery thermal management for electric vehicles with battery degradation minimization

The control of a battery thermal management system (BTMS) is essential for the thermal safety, energy efficiency, and durability of electric vehicles (EVs) in hot weather. To address the battery cooling optimization problem, this paper utilizes dynamic programming (DP) to develop an online rule-based control strategy. Firstly, an electrical–thermal-aging model of the LiFePO4 battery pack is established. A control-oriented onboard BTMS model is proposed and verified under different speed profiles and temperatures. Then in the DP framework, a cost function consisting of battery aging cost and cooling-induced electricity cost is minimized to obtain the optimal compressor power. By exacting three rules ”fast cooling, slow cooling, and temperature-maintaining” from the DP result, a near-optimal rule-based cooling strategy, which uses as much regenerative energy as possible to cool the battery pack, is proposed for online execution. Simulation results show that the proposed online strategy can dramatically improve the driving economy and reduce battery degradation under diverse operation conditions, achieving less than a 2.18% difference in battery loss compared to the offline DP. Recommendations regarding battery cooling under different real-world cases are finally provided.

Optimal Scheduling in IoT-Driven Smart Isolated Microgrids Based on Deep Reinforcement Learning

In this paper, we investigate the scheduling issue of diesel generators (DGs) in an Internet of Things (IoT)-Driven isolated microgrid (MG) by deep reinforcement learning (DRL). The renewable energy is fully exploited under the uncertainty of renewable generation and load demand. The DRL agent learns an optimal policy from history renewable and load data of previous days, where the policy can generate real-time decisions based on observations of past renewable and load data of previous hours collected by connected sensors. The goal is to reduce operating cost on the premise of ensuring supply-demand balance. In specific, a novel finite-horizon partial observable Markov decision process (POMDP) model is conceived considering the spinning reserve. In order to overcome the challenge of discrete-continuous hybrid action space due to the binary DG switching decision and continuous energy dispatch (ED) decision, a DRL algorithm, namely the hybrid action finite-horizon RDPG (HAFH-RDPG), is proposed. HAFH-RDPG seamlessly integrates two classical DRL algorithms, i.e., deep Q-network (DQN) and recurrent deterministic policy gradient (RDPG), based on a finite-horizon dynamic programming (DP) framework. Extensive experiments are performed with real-world data in an IoT-driven MG to evaluate the capability of the proposed algorithm in handling the uncertainty due to inter-hour and inter-day power fluctuation and to compare its performance with those of the benchmark algorithms.