Differential Dynamic Programming Research Articles

Contact-implicit Model Predictive Control: Controlling diverse quadruped motions without pre-planned contact modes or trajectories

This paper presents a contact-implicit model predictive control (MPC) framework for the real-time discovery of multi-contact motions, without predefined contact mode sequences or foothold positions. This approach utilizes the contact-implicit differential dynamic programming (DDP) framework, merging the hard contact model with a linear complementarity constraint. We propose the analytical gradient of the contact impulse based on relaxed complementarity constraints to further the exploration of a variety of contact modes. By leveraging a hard contact model-based simulation and computation of search direction through a smooth gradient, our methodology identifies dynamically feasible state trajectories, control inputs, and contact forces while simultaneously unveiling new contact mode sequences. However, the broadened scope of contact modes does not always ensure real-world applicability. Recognizing this, we implemented differentiable cost terms to guide foot trajectories and make gait patterns. Furthermore, to address the challenge of unstable initial roll-outs in an MPC setting, we employ the multiple shooting variant of DDP. The efficacy of the proposed framework is validated through simulations and real-world demonstrations using a 45 kg HOUND quadruped robot, performing various tasks in simulation and showcasing actual experiments involving a forward trot and a front-leg rearing motion.

Two-stage dynamic real-time optimization framework using parameter-dependent differential dynamic programming

The purpose of chemical process control includes proactive adjustment of the operation to make the most profit out of it. Within this context, real-time optimization (RTO) is proposed and extended to dynamic RTO (DRTO) in the hierarchical control structure, usually having model predictive control (MPC) below. However, online tractability confined the model complexity of RTO and MPC, which results in model inconsistency and, even, incompatible solutions. Here we use parameter-dependent differential dynamic programming (PDDP) to incorporate the closed-loop behavior of the controller in an RTO layer to reduce problem complexity and online computation time. The adaptive control performance of PDDP and the efficacy of closed-loop DRTO formulation with PDDP is demonstrated with the reaction-storage-separation network system control. Consequently, PDDP provides a useful parameterization method to express closed-loop system dynamics, which enables fast feedback control and integrated plant optimization.

A learning-based model predictive control scheme for injection speed tracking in injection molding process

Injection molding is a pivotal industrial process renowned for its high production speed, efficiency, and automation. Controlling the motion speed of injection molding machines is a crucial factor that influences production processes, directly affecting product quality and efficiency. This paper aims to tackle the challenge of achieving optimal tracking control of injection speed in a standard class of injection molding machines (IMMs) characterized by nonlinear dynamics. To achieve this goal, we propose a learning-based model predictive control (LMPC) scheme that incorporates Gaussian process regression (GPR) to predict and model uncertainty in the injection molding process (IMP). Specifically, the scheme formulates a nonlinear tracking control problem for injection speed, utilizing a GPR-based learning residual model to capture uncertainty and provide accurate predictions. It learns the dynamics model and historical data of the IMM, automatically adjusting the injection speed according to target requirements for optimal production control. Additionally, the optimization problem is efficiently solved using a control-constrained differential dynamic programming approach. Finally, we conduct comprehensive numerical experiments to demonstrate the effectiveness and efficiency of the proposed LMPC scheme for controlling injection speed in IMP.

Observability-Aware Differential Dynamic Programming with Impulsive Maneuvers

In autonomous space systems, the reliability of navigation systems is essential. Observability in autonomous orbit determination techniques depends on the spacecraft’s orbital motion, making the design of autonomous navigation systems and orbital maneuvers a coupled process. This study develops a stable and efficient algorithm based on differential dynamic programming to design maneuver sequences that improve navigation performance. Our approach incorporates the Fisher information matrix into a cost function to quantify state observability and facilitates its convergence using a semi-analytic gradient and Hessian derived under impulsive maneuvers. Two numerical examples show the validity and effectiveness of our algorithm. The results indicate the stability and efficiency in determining maneuver sequences and the improvement of state estimation accuracy along an optimized trajectory. It is also applicable to other observability-aware optimal control problems because the algorithm is independent of specific systems.

Constrained Parameterized Differential Dynamic Programming for Waypoint-Trajectory Optimization

Unmanned aerial vehicles (UAVs) are required to pass through multiple important waypoints as quickly as possible in courier delivery, enemy reconnaissance and other tasks to eventually reach the target position. There are two important problems to be solved in such tasks: constraining the trajectory to pass through intermediate waypoints and optimizing the flight time between these waypoints. A constrained parameterized differential dynamic programming (C-PDDP) algorithm is proposed for meeting multiple waypoint constraints and free-time constraints between waypoints to deal with these two issues. By considering the intermediate waypoint constraints as a kind of path state constraint, the penalty function method is adopted to constrain the trajectory to pass through the waypoints. For the free-time constraints, the flight times between waypoints are converted into time-invariant parameters and updated at the trajectory instants corresponding to the waypoints. The effectiveness of the proposed C-PDDP algorithm under waypoint constraints and free-time constraints is verified through numerical simulations of the UAV multi-point reconnaissance problem with five different waypoints. After comparing the proposed algorithm with fixed-time constrained DDP (C-DDP), it is found that C-PDDP can optimize the flight time of the trajectory with three segments to 7.35 s, 9.50 s and 6.71 s, respectively. In addition, the maximum error of the optimized trajectory waypoints of the C-PDDP algorithm is 1.06 m, which is much smaller than that (7 m) of the C-DDP algorithm used for comparison. A total of 500 Monte Carlo tests were simulated to demonstrate how the proposed algorithm remains robust to random initial guesses.

Uncertainty-resilient constrained rendezvous trajectory optimization via stochastic feedback control and unscented transformation

This paper proposes an innovative approach for constrained rendezvous trajectory optimization (CRTO) using stochastic optimal feedback control and unscented transform (UT) uncertainty quantification. The method overcomes limitations of traditional deterministic CRTO solutions that suffer from uncontrolled terminal state error growth and severely deteriorated path constraint satisfaction when initial state uncertainty, dynamics uncertainty, and navigation uncertainty are considered. The approach involves constructing a stochastic optimal feedback control (SOFC) problem with chance constraints and introducing linear feedback control to regulate both the mean and variance of terminal state errors. UT is employed to approximately quantify the state errors and their propagation, transforming the SOFC problem into an unconstrained deterministic optimization (UDO) problem. Differential dynamic programming (DDP) is then used to solve the UDO problem. The obtained stochastic optimal solution provides a robust rendezvous trajectory and a corresponding explicit closed-loop guidance law, improving terminal accuracy and path constraint satisfaction in the presence of various considered uncertainties. The influence of different term weights in the objective function on terminal accuracy and path constraint satisfaction is also studied. The effectiveness of the proposed approach is verified through Monte Carlo simulation, demonstrating the robustness of the closed-loop control strategy. The results highlight the potential of the method in enhancing terminal accuracy and path constraint satisfaction in uncertain rendezvous scenarios.

Distributed Differential Dynamic Programming Architectures for Large-Scale Multiagent Control

Distributed Differential Dynamic Programming Architectures for Large-Scale Multiagent Control

Modified dynamic programming algorithms for GLOSA systems with stochastic signal switching times

A discrete-time stochastic optimal control problem was recently proposed to address the GLOSA (Green Light Optimal Speed Advisory) problem in cases where the next signal switching time is decided in real time and is therefore uncertain in advance. The corresponding numerical solution via SDP (Stochastic Dynamic Programming) calls for substantial computation time, which excludes problem solution in the vehicle’s on-board computer in real time. In this context, the present paper concentrates on the challenge of developing numerical algorithms to solve efficiently the stochastic GLOSA problem. As a first attempt to overcome the computation time bottleneck, a modified version of Dynamic Programming, known as Discrete Differential Dynamic Programming (DDDP) was recently employed for the numerical solution of the stochastic optimal control problem and was demonstrated to achieve results equivalent to those obtained with the ordinary SDP algorithm, albeit with significantly reduced computation times. After an outline of the DDDP approach, the present work considers a second modified version of Dynamic Programming, known as Differential Dynamic Programming (DDP). For the stochastic GLOSA problem, it is demonstrated that DDP achieves quasi-instantaneous (extremely fast) solutions in terms of CPU times, which allows for the proposed approach to be readily executable online, in an MPC (Model Predictive Control) framework, in the vehicle’s on-board computer. The novel numerical approach is tested and demonstrated by use of realistic examples and is compared to the SDP and DDDP solutions. It should be noted that DDP does not require discretization of variables, hence the obtained solutions may be slightly superior to the standard SDP solutions.

Fourier–Hermite Dynamic Programming for Optimal Control

In this paper, we propose a novel computational method for solving non-linear optimal control problems. The method is based on the use of Fourier–Hermite series for approximating the action-value function arising in dynamic programming instead of the conventional Taylor-series expansion used in differential dynamic programming (DDP). The coefficients of the Fourier–Hermite series can be numerically computed by using sigma-point methods, which leads to a novel class of sigma-point based dynamic programming methods. We also prove the quadratic convergence of the method and experimentally test its performance against other methods.

Fixed point theorems of enriched multivalued mappings via sequentially equivalent Hausdorff metric

Abstract Recently, Abbas et al. [Enriched multivalued contractions with applications to differential inclusions and dynamic programming, Symmetry 13(8) (2021), 1350] obtained an interesting generalization of the Nadler fixed point theorem by introducing the concept of enriched multivalued contraction in the framework of Banach spaces. In this article, we define a new class of metrics on the family of closed and bounded subsets of a given metric space. Furthermore, fixed point theorems were established for enriched multi-valued contractions by substituting the Hausdorff metric with metrics from a specific class that are either metrically or sequentially equivalent to the Hausdorff metric. Some examples are provided to illustrate the concepts and results presented herein. These results improve, unify, and generalize several comparable results in the literature.

Differential dynamic programming for finite‐horizon zero‐sum differential games of nonlinear systems

AbstractIn this article, we present an iterative algorithm based on differential dynamic programming (DDP) for finite‐horizon two‐person zero‐sum differential games. The technique of DDP is used to expand the Hamilton–Jacobi–Isaacs (HJI) partial differential equation into higher‐order differential equations. Using value function and saddle point approximations, the DDP expansion is transformed into algebraic matrix equation in integral form. Based on the algebraic matrix equation, a DDP iterative algorithm is developed to learn the solution to the differential games. Strict proof is proposed to guarantee the iterative convergences of the value function and saddle point. The new algorithm is fundamentally different from existing results, in the sense that it overcome the technical obstacle to address the time‐varying behavior of HJI partial differential equation. Simulation examples are given to demonstrate the effectiveness of the proposed method.

Robust Sampling-Based Control of Mobile Manipulators for Interaction With Articulated Objects

In this article, we investigate and deploy sampling-based control techniques for the challenging task of the mobile manipulation of articulated objects. By their nature, manipulation tasks necessitate environment interactions, which require the handling of nondifferentiable switching contact dynamics. These dynamics represent a strong limitation for traditional gradient-based optimization methods, such as model-predictive control and differential dynamic programming, which often rely on heuristics for trajectory generation. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Sampling-based</i> techniques alleviate these constraints but do not ensure robots' stability and input/state constraints either. On the other hand, real-world applications in human environments require safety and robustness to unexpected events. For this reason, we propose a novel framework for safe robotic manipulation of movable articulated objects. The framework combines sampling-based control together with <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">control barrier functions</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">passivity theory</i> that, thanks to formal stability guarantees, enhance the safety and robustness of the method. We also provide the practical insights that enable robust deployment of stochastic control using a conventional central processing unit. We deploy the algorithm on a ten-degree-of-freedom mobile manipulator robot. Finally, we open source our generic and multithreaded implementation.

Analytical differentiation of the articulated-body algorithm: a geometric multilinear approach

This paper presents a new geometric algorithm to efficiently compute the analytical differentiation of the Articulated-body Algorithm (ABA) with respect to the state variables. Despite the fact that ABA solves the forward-dynamics problem in linear time for branched-multibody systems, its explicit differentiation is not straightforward, and computationally demanding tensor products and contractions appear. The proposed algorithm makes use of Lie algebraic and multilinear operations to recursively exploit the underlying sparsity of the linearization problem. As a result, the arithmetic complexity is dramatically reduced without affecting the analytical solution. Since the linealization of forward dynamics is of great importance in robot-trajectory optimization, a differential dynamic programming solver is employed to demonstrate the performance of our algorithm with humanoid robot models such as NAO and HRP-2. In addition, we provide the computational cost with different robots using an optimized C++ implementation that can be found at https://github.com/garechav/geombd_crtp.git .

Model Predictive Control of Autonomous Vehicles With Integrated Barriers Using Occupancy Grid Maps

Nonlinear model predictive control (NMPC) is an efficient and proven method for optimization-based autonomous vehicle motion planning. Among the various approaches, the iterative linear quadratic regulator, a differential dynamic programming variant, is a well-known efficient nonlinear optimization method. In safety-critical control systems, controllers should address inequality-constrained optimization problems. In this letter, we design a single unified constraint using an occupancy grid map. We convert this inequality-constrained optimization problem into an unconstrained optimization problem by appending a single integrated discrete barrier state to the system model. This approach simplifies complex motion planning problems and reduces computational costs. In the proposed method, we first discretize the surrounding environment with an occupancy grid map and design a single constraint that ensures that only cells with values less than a predefined threshold can be traversed by the ego vehicle. Then, we define a single integrated discrete barrier state to introduce this constraint into the motion planning algorithm. The proposed method, a penalty method, and the augmented Lagrangian method are tested on a real-time software-in-the-loop simulation using CarMaker and ROS. The simulation results of pop-up obstacle avoidance scenarios show the benefits of the proposed method, such as reduced time costs and increased robustness.

Second-Order Differential Dynamic Programming for Whole-Body MPC of Legged Robots

This paper presents the first use-case of full second-order Differential Dynamic Programming (DDP) with a whole-body model for model predictive control on legged robot hardware. Recent advances in the literature show that DDP can be simplified by exploiting the dynamics structure in the algorithm, allowing the use of reverse-mode derivative accumulation to efficiently compute dynamics sensitives. These advances allow DDP to be run at similar compute times as the iterative LQR (iLQR) algorithm, its first-order counterpart. Beyond a hardware implementation of these past theoretical developments, this paper provides a characterization of DDP vs. iLQR following push disturbances. One challenge is that DDP often requires regularization to ensure the algorithm improves its control law from iteration to iteration, and an expensive matrix reconditioning is often necessary. We explore a novel methodology to carry out this matrix reconditioning, which considers the local geometry of the cost landscape. The resulting controller is shown to achieve lower costs and withstand larger disturbances as compared to traditional regularization schemes.

Distributionally Robust Differential Dynamic Programming With Wasserstein Distance

Differential dynamic programming (DDP) is a popular technique for solving nonlinear optimal control problems with locally quadratic approximations. However, existing DDP methods are not designed for stochastic systems with unknown disturbance distributions. To address this limitation, we propose a novel DDP method that approximately solves the Wasserstein distributionally robust control (WDRC) problem, where the true disturbance distribution is unknown but a disturbance sample dataset is given. Our approach aims to develop a practical and computationally efficient DDP solution. To achieve this, we use the Kantrovich duality principle to decompose the value function in a novel way and derive closed-form expressions of the distributionally robust control and worst-case distribution policies to be used in each iteration of our DDP algorithm. This characterization makes our method tractable and scalable without the need for numerically solving any minimax optimization problems. The superior out-of-sample performance and scalability of our algorithm are demonstrated through kinematic car navigation and coupled oscillator problems.

Accelerating Hybrid Systems Differential Dynamic Programming

Abstract This letter presents approaches that reduce the computational demand of including second-order dynamics sensitivity information into optimization algorithms for robots in contact with the environment. A full second-order differential dynamic programming (DDP) algorithm is presented where all the necessary dynamics partial derivatives are computed with the same complexity as DDP’s first-order counterpart, the iterative linear quadratic regulator (iLQR). Compared to linearized models used in iLQR, DDP more accurately represents the dynamics locally, but it is not often used since the second-order partials of the dynamics are tensorial and expensive to compute. This work illustrates how to avoid the need for computing the derivative tensor by instead leveraging reverse-mode accumulation of derivatives, extending previous work for unconstrained systems. We exploit the structure of the contact-constrained dynamics in this process. The performance of the proposed approaches is benchmarked with a simulated model of the MIT Mini Cheetah executing a bounding gait.

Data‐Driven Optimal Tracking Control of Nonlinear Affine Systems Based on Differential Dynamic Programming

In this paper, a data‐driven differential dynamic programming (DDP) algorithm is presented for the optimal tracking problems of nonlinear affine systems with unknown drift dynamics. Optimal tracking dynamics are established between the system states and expected trajectory. By using the DDP method and test data, a second‐order data‐driven framework of the Hamilton–Jacobi–Bellman (HJB) equation is constructed, in which system approximation and value function approximation are used. By using this framework, a data‐driven iteration algorithm is proposed to achieve the optimal tracking controller. A simulation example is provided to verify the effectiveness of the proposed approach.

Flexible Final-Time Stochastic Differential Dynamic Programming for Autonomous Vehicle Trajectory Optimization

In this paper, the problem of autonomous vehicle trajectory optimization with flexible final time is concerned under the consideration of stochastic disturbances. Stochastic differential dynamic programming (SDDP) has been widely applied to address this type of problem due to its fast convergence and capability to handle model errors. Typically, traditional SDDP is designed with a deterministic final time which is mainly assigned based on expert experiences. However, this may limit the implementation of this approach. To deal with this issue, we define the final time as a new optimal variable and propose an enhanced version of SDDP, named flexible final time SDDP (FFT-SDDP). The stochastic dynamic system is then characterized as a deterministic one with random state perturbances. An unscented transform approach is utilised to cope with the perturbed expected values. To verify the effectiveness of the proposed approach, a three-dimensional missile trajectory optimization problem is tested as an example. The simulation results show that the proposed method is able to address the stochastic trajectory optimization problem and can provide stronger robustness compared to other algorithms.

Fast, Smooth, and Safe: Implicit Control Barrier Functions Through Reach-Avoid Differential Dynamic Programming

Fast, Smooth, and Safe: Implicit Control Barrier Functions Through Reach-Avoid Differential Dynamic Programming