Stochastic Differential Dynamic Programming Research Articles

Flexible Final-Time Stochastic Differential Dynamic Programming for Autonomous Vehicle Trajectory Optimization

In this paper, the problem of autonomous vehicle trajectory optimization with flexible final time is concerned under the consideration of stochastic disturbances. Stochastic differential dynamic programming (SDDP) has been widely applied to address this type of problem due to its fast convergence and capability to handle model errors. Typically, traditional SDDP is designed with a deterministic final time which is mainly assigned based on expert experiences. However, this may limit the implementation of this approach. To deal with this issue, we define the final time as a new optimal variable and propose an enhanced version of SDDP, named flexible final time SDDP (FFT-SDDP). The stochastic dynamic system is then characterized as a deterministic one with random state perturbances. An unscented transform approach is utilised to cope with the perturbed expected values. To verify the effectiveness of the proposed approach, a three-dimensional missile trajectory optimization problem is tested as an example. The simulation results show that the proposed method is able to address the stochastic trajectory optimization problem and can provide stronger robustness compared to other algorithms.

Optimal Position and Attitude Control of Quadcopter Using Stochastic Differential Dynamic Programming with Input Saturation Constraints

This paper considers the position and attitude control of a quadcopter in the presence of stochastic disturbances. Basic quadcopter dynamics is modeled as a nonlinear stochastic system described by a stochastic differential equation. Subsequently, the position and attitude control is formulated as a nonlinear stochastic optimal control problem with input saturation constraints. To solve this problem, a continuous-time stochastic differential dynamic programming (DDP) method with input saturation constraints is newly proposed. Finally, numerical simulations demonstrate the effectiveness of the proposed method by comparing it with the linear quadratic Gaussian and the deterministic DDP with input saturation constraints.

Uncertainty-Constrained Differential Dynamic Programming in Belief Space for Vision Based Robots

Most mobile robots follow a modular sense-plan-act system architecture that can lead to poor performance or even catastrophic failure for visual inertial navigation systems due to trajectories devoid of feature matches. Planning in belief space provides a unified approach to tightly couple the perception, planning and control modules, leading to trajectories that are robust to noisy measurements and disturbances. However, existing methods handle uncertainties as costs that require manual tuning for varying environments and hardware. We therefore propose a novel trajectory optimization formulation that incorporates inequality constraints on uncertainty and a novel Augmented Lagrangian based stochastic differential dynamic programming method in belief space. Furthermore, we develop a probabilistic visibility model that accounts for discontinuities due to feature visibility limits. Our simulation tests demonstrate that our method can handle inequality constraints in different environments, for holonomic and nonholonomic motion models with no manual tuning of uncertainty costs involved. We also show the improved optimization performance in belief space due to our visibility model.

Approximate Stochastic Differential Dynamic Programming for Hybrid Vehicle Energy Management

This paper develops a new computational approach for energy management in a hydraulic hybrid vehicle. The developed algorithm, called approximate stochastic differential dynamic programming (ASDDP) is a variant of the classic differential dynamic programming algorithm. The simulation results are discussed for two Environmental Protection Agency drive cycles and one real world cycle based on collected data. Flexibility of the ASDDP algorithm is demonstrated as real-time driver behavior learning, and forecasted road grade information are incorporated into the control setup. Real-time potential of ASDDP is evaluated in a hardware-in-the-loop (HIL) experimental setup.

Stochastic Differential Dynamic Programming with Unscented Transform for Low-Thrust Trajectory Design

Low-thrust propulsion is a key technology for space exploration, and much work in astrodynamics has focused on the mathematical modeling and the optimization of low-thrust trajectories. Typically, a nominal trajectory is designed in a deterministic system. To account for model and execution errors, mission designers heuristically add margins, for example, by reducing the thrust and specific impulse or by computing penalties for specific failures. These conventional methods are time-consuming, done by hand by experts, and lead to conservative margins. This paper introduces a new method to compute nominal trajectories, taking into account disturbances. The method is based on stochastic differential dynamic programming, which has been used in the field of reinforcement learning but not yet in astrodynamics. A modified version of stochastic differential dynamic programming is proposed, where the stochastic dynamical system is modeled as the deterministic dynamical system with random state perturbations, the perturbed trajectories are corrected by linear feedback control policies, and the expected value is computed with the unscented transform method, which enables solving trajectory design problems. Finally, numerical examples are presented, where the solutions of the proposed method are more robust to errors and require fewer penalties than those computed with traditional approaches, when uncertainties are introduced.

Stochastic differential dynamic programming for multi-reservoir system control

: As with all dynamic programming formulations, differential dynamic programming (DDP) successfully exploits the sequential decision structure of multi-reservoir optimization problems, overcomes difficulties with the nonconvexity of energy production functions for hydropower systems, and provides optimal feedback release policies. DDP is particularly well suited to optimizing large-scale multi-reservoir systems due to its relative insensitivity to state-space dimensionality. This advantage of DDP encourages expansion of the state vector to include additional multi-lag hydrologic information and/or future inflow forecasts in developing optimal reservoir release policies. Unfortunately, attempts at extending DDP to the stochastic case have not been entirely successful. A modified stochastic DDP algorithm is presented which overcomes difficulties in previous formulations. Application of the algorithm to a four-reservoir hydropower system demonstrates its capabilities as an efficient approach to solving stochastic multi-reservoir optimization problems. The algorithm is also applied to a single reservoir problem with inclusion of multi-lag hydrologic information in the state vector. Results provide evidence of significant benefits in direct inclusion of expanded hydrologic state information in optimal feedback release policies.