Optimal Control Policy Research Articles

Discrete-Time Hybrid Control Processes with Unbounded Costs

This paper extends the results provided in Jasso-Fuentes et al. (Appl Math Optim 81(2):409–441, 2020b) and Jasso-Fuentes et al. (Pure Appl Funct Anal 9(3):675–704, 2024) regarding the study of discrete-time hybrid stochastic models with general spaces and total discounted payoffs. This extension incorporates the handling of negative and/or unbounded costs per stage. In particular, it encompasses interesting applications, such as scenarios where the controller optimizes net costs, social welfare costs, or distances between points. These situations arise when assumptions of both non-negativeness and boundedness on the cost per stage do not apply. Our proposal relies on Lyapunov-like conditions, enabling, among other aspects, the finiteness of the value function and the existence of solutions to the associated dynamic programming equation. This equation is crucial for deriving optimal control policies. To illustrate our theory, we include an example in inventory-manufacturing management, highlighting its evident hybrid nature.

Discrete-time hybrid control with risk-sensitive discounted costs

AbstractThis paper studies discrete-time hybrid control problems where the controller’s payoff function follows a risk-sensitive discounted model. The associated discount factor can depend on state and action variables, which can even take values of zero or one. We prove the existence of optimal control policies under two different hypotheses. The following technique is the dynamic programming method, where we characterize the minimum cost as the solution of the so-called dynamic programming equations and find the above optimal policies through these equations. We illustrate our theory with practical examples involving inventory-manufacturing systems, and pollution management.

Reinforcement Learning for Fuselage Shape Control during Aircraft Assembly

Critical safety requirements necessitate ultra-high precision quality control during the assembly of large aerospace components to reduce the mismatch between parts to be joined. Traditional methods use heuristic shape adjustment or surrogate model-based control. These methods are limited by reliance on accurate model learning and inadequate robustness to varying initial assembly conditions. To address these limitations, this paper proposes a model-free reinforcement learning approach for adaptive fuselage shape control during aircraft assembly. The trained reinforcement learning agent directly adjusts the aircraft components in response to their part variations and enables an autonomous system (like AlphaGo) to learn the optimal shape control policy. Specifically, the reinforcement learning environment is built on the finite element simulator. A reward function is developed to capture the optimization objective and introduces a scheme to enforce the original constraints. The proximal policy optimization algorithm is modified to speed up the learning progress and achieve better final performance. In the case study, the root-mean-square gap between components is reduced by 98.4% on average compared with their initial shape mismatch. Our proposed method outperforms the benchmark methods with smaller final shape errors, smaller maximum forces, and lower variations across different test samples.

Variability, Attributes, and Drivers of Optimal Forecast-Informed Reservoir Operating Policies for Water Supply and Flood Control in California

Variability, Attributes, and Drivers of Optimal Forecast-Informed Reservoir Operating Policies for Water Supply and Flood Control in California

Optimization of geological carbon storage operations with multimodal latent dynamic model and deep reinforcement learning

Identifying the time-varying control schemes that maximize storage performance is critical to the commercial deployment of geological carbon storage (GCS) projects. However, the optimization process typically demands extensive resource-intensive simulation evaluations, which poses significant computational challenges and practical limitations. In this study, we presented the multimodal latent dynamic (MLD) model, a novel deep learning framework for fast flow prediction and well control optimization in GCS operations. The MLD model implicitly characterizes the forward compositional simulation process through three components: a representation module that learns compressed latent representations of the system, a transition module that approximates the evolution of the system states in the low-dimensional latent space, and a prediction module that forecasts the flow responses for given well controls. A novel model training strategy combining a regression loss and a joint-embedding consistency loss was introduced to jointly optimize the three modules, which enhances the temporal consistency of the learned representations and ensures multi-step prediction accuracy. Unlike most existing deep learning models designed for systems with specific parameters, the MLD model supports arbitrary input modalities, thereby enabling comprehensive consideration of interactions between diverse types of data, including dynamic state variables, static spatial system parameters, rock and fluid properties, as well as external well settings. Since the MLD model mirrors the structure of a Markov decision process (MDP) that computes state transitions and rewards (i.e., economic calculation for flow responses) for given states and actions, it can serve as an interactive environment to train deep reinforcement learning agents. Specifically, the soft actor-critic (SAC) algorithm was employed to learn an optimal control policy that maximizes the net present value (NPV) from the experiences gained by continuous interactions with the MLD model. The efficacy of the proposed approach was first compared against commonly used simulation-based evolutionary algorithm and surrogate-assisted evolutionary algorithm on a deterministic GCS optimization case, showing that the proposed approach achieves the highest NPV, while reducing the required computational resources by more than 60%. The framework was further applied to the generalizable GCS optimization case. The results indicate that the trained agent is capable of harnessing the knowledge learned from previous scenarios to provide improved decisions for newly encountered scenarios, demonstrating promising generalization performance.

Optimal Control of a Semi-Active Suspension System Collaborated by an Active Aerodynamic Surface Based on a Quarter-Car Model

This paper addresses the trade-off between ride comfort and road-holding capability of a quarter-car semi-active suspension system, collaborated by an active aerodynamic surface (AAS), using an optimal control policy. The semi-active suspension system is more practical to implement due to its low energy consumption than the active suspension system while significantly improving ride comfort. First, a model of the two-DOF quarter-car semi-active suspension in the presence of an active airfoil with two weighting sets based on ride comfort and road-holding preferences is presented. Then, a comprehensive comparative study of the improved target performance indices with various suspension systems is performed to evaluate the proposed suspension performance. Time-domain and frequency-domain analyses are conducted in MATLAB® (R2024a). From the time-domain analysis, the total performance measure is enhanced by about 50% and 35 to 45%, respectively, compared to passive and active suspension systems. The results demonstrate that a semi-active suspension system with an active aerodynamic control surface simultaneously improves the conflicting target parameters of passenger comfort and road holding. Utilizing the aerodynamic effect, the proposed system enhances the vehicle’s dynamic stability and passenger comfort compared to other suspension systems.

Reinforcement Learning for Jointly Optimal Coding and Control Policies for a Markovian System Controlled over a Communication Channel

This paper develops approximation and optimality results for the optimal control of a networked system. In this system, a Markovian process is managed over a finite-rate, noiseless communication channel. Solving this problem involves determining joint optimal coding and control policies that minimize cost over time. While theoretical results on optimal coding and control structures exist, practical implementation has remained largely infeasible for non-linear systems due to the computational complexities and uncountable state spaces. This research introduces an approach that combines structural results with reinforcement learning (RL) techniques. The method uses regularity properties of the system to approximate the uncountable state space with a countable one, which allows reinforcement learning algorithms to achieve near-optimal solutions. Specifically, we establish that finite model approximations (where infinite state spaces are quantized to finite ones) and sliding finite window approximations (where a finite memory "window" of past control actions is maintained at each time step) can be employed to develop near-optimality. These approximations allow the system to be reformulated as a Markov Decision Problem (MDP) with a finite state space, making RL algorithms implementable. The convergence of the reinforcement learning algorithm to a near-optimal policy (under both the previous approximations) is supported by theoretical analysis and performance simulations. We ensure that the solutions obtained are not only computationally feasible but also nearly optimal with respect to the original problem. The applications of this work extend to many networked control systems, especially those with zero-delay coding and partially observable Markov decision processes (POMDPs). By integrating structural results with learning algorithms, this paper provides a practical framework for implementing optimal control in finite-rate environments.

Experimental Implementation of a TD3 Agent Based Speed Controller for Direct Torque Control of PMSM Drives

This manuscript presents a novel control approach for Permanent Magnet Synchronous Motors (PMSMs) by integrating the widely used Direct Torque Control (DTC) method with Deep Reinforcement Learning (DRL), specifically the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm. The TD3 algorithm is well-suited to address the challenges posed by high-dimensional state and action spaces in PMSM drives. The conventional Proportional–Integral (PI) controller in the outer loop of DTC is replaced with the DRL based TD3 agent to overcome the limitations of traditional PI controllers, such as model dependency and complex parameter tuning. The DRL technique allows the agent to learn optimal control policies from experience, making it suitable for complex and nonlinear control problems. Extensive simulation studies demonstrate the effectiveness of the TD3-based control in achieving accurate speed tracking under varying operating conditions. Real-time experimental validation using a TMS320F28379D digital signal processor confirms the feasibility of the proposed DRL based approach. The research offers new insights into improving the performance of PMSM drives and paves the way for future advancements in electric drive control.

ADP-based online compensation hierarchical sliding-mode control for partially unknown switched nonlinear systems with actuator failures

This article investigates an adaptive dynamic programming-based online compensation hierarchical sliding-mode control problem for a class of partially unknown switched nonlinear systems with actuator failures and uncertain perturbations under an identifier-critic neural networks architecture. Firstly, by introducing a cost function related to hierarchical sliding-mode surfaces for the nominal system, the original control problem is equivalently converted into an optimal control problem. To obtain this optimal control policy, the Hamilton–Jacobi–Bellman equation is solved through an adaptive dynamic programming method. Compared with conventional adaptive dynamic programming methods, the identifier-critic network architecture not only overcomes the limitation on the unknown internal dynamic but also eliminates the approximation error arising from the actor network. The weights in the critic network are tuned via the gradient descent approach and the experience replay technology, such that the persistence of excitation condition can be relaxed. Then, a compensation term containing hierarchical sliding-mode surfaces is used to offset uncertain actuator failures without the fault detection and isolation unit. Based on the Lyapunov stability theory, all states of the closed-loop nonlinear system are stable in the sense of uniformly ultimately boundedness. Finally, numerical and practical examples are given to demonstrate the effectiveness of our presented online compensation control strategy.

Robust Risk Control with Reinsurance and CAT Bonds

This research studies robust risk control policies for an ambiguity-averse insurer. The insurer can manage its risk exposure through the purchase of reinsurance and the issuance of parametric catastrophe (CAT) bonds, which are linked to an exogenous trigger index. The insurer worries about the veracity of the model and seeks to develop a robust strategy to minimize the discounted ruin probability. We allow the insurer to exhibit different levels of ambiguity aversion toward its own claims and the trigger index. By employing a robust control approach, we analytically derive the optimal risk control policies that provide the insurer with guidelines on how to effectively manage its risk in an ambiguous environment. We present numerical examples that showcase scenarios in which CAT bonds can be utilized as an effective risk mitigation tool. Furthermore, we assess the potential welfare losses that could arise if the insurer fails to account for model uncertainty or lacks the ability to issue CAT bonds.

Novel generalized policy iteration for efficient evolving control of nonlinear systems

In this article, we construct a novel generalized policy iteration framework to address optimal regulation problems for discrete-time nonlinear systems in a more efficient way. Relevant properties are investigated for the framework, including monotonicity and convergence of the iterative value function sequence as well as the admissibility of the iterative control policy. Additionally, an innovative approach is developed to seek an initial admissible control policy for the framework with an adjustable searching speed. Based on these, an evolving control algorithm is presented with stability guarantee. This algorithm employs iterative control policies for system control during the computation of the optimal control policy, as opposed to waiting for the generation of the optimal control policy before implementing control. Eventually, two simulation experiments are conducted with real-world physical backgrounds, in order to illustrate the performance of the proposed strategy.

Trajectory tracking control of vectored thruster autonomous underwater vehicles based on deep reinforcement learning

ABSTRACT Vectored thruster autonomous underwater vehicles (AUVs) offer superior manoeuvrability compared to traditional fin and rudder systems, especially at low or zero velocities. However, controlling these AUVs becomes challenging in the presence of unpredictable interference forces and external water flow velocities. This paper introduces a novel three-dimensional trajectory tracking method for vectored thruster AUVs using reinforcement learning, enabling the system to learn optimal control policies through environmental interaction. A 5-degree of freedom (5-DOF) model is developed from the kinematics and dynamics equations of the underactuated AUV. An extended state observer (ESO) is used to estimate the differential expansion state based on observed variables. A reinforcement learning-based trajectory tracking strategy is then implemented. Simulation experiments demonstrate the method’s effectiveness in precisely controlling AUVs under varying environmental conditions, achieving exceptional tracking accuracy even in the presence of random interference forces and external water flow velocities.

Online optimal tracking control of unknown nonlinear singularly perturbed systems using single network adaptive critic with improved learning

This study is the first time devoted to seek an online optimal tracking solution for unknown nonlinear singularly perturbed systems based on single network adaptive critic (SNAC) design. Firstly, a novel identifier with more efficient parametric multi-time scales differential neural network (PMTSDNN) is developed to obtain the unknown system dynamics. Then, based on the identification results, the online optimal tracking controller consists of an adaptive steady control term and an optimal feedback control term is developed by using SNAC to solve the Hamilton–Jacobi–Bellman (HJB) equation online. New learning law considering filtered parameter identification error is developed for the PMTSDNN identifier and the SNAC, which can realize online synchronous learning and fast convergence. The Lyapunov approach is synthesized to ensure the convergence characteristics of the overall closed loop system consisting of the PMTSDNN identifier, the SNAC and the optimal tracking control policy. Three examples are provided to illustrate the effectiveness of the investigated method.

Swarm‐intelligence‐based coordinated control of electric heatings for voltage stabilization with zero communication burden

AbstractThe movement towards electrification is entailing deep changes in power systems. On the demand side, the adoption of electric heating (EH) has grown rapidly in recent years. To eliminate the voltage fluctuations caused by the random switching‐on/‐off behaviors of EHs in some application scenes with special‐power‐line in low voltage distribution networks (LVDNs), a swarm‐intelligence‐based coordinated control approach of EHs for voltage stabilization with zero communication burden while guaranteeing the users’ thermal comforts is researched. A novel bird‐perch‐on‐branch (BPB) ‐based swarm intelligence theory is proposed, which reveals the mapping relation between local observations and global states. On this theoretical basis, a multi‐agent reinforcement learning (MARL) framework is developed for the coordinated dispatch of EHs, where the deep‐Q‐network (DQN) algorithm is adopted to learn and generate the optimal control policy for each EH. The implementation of the proposed MADQN‐BPB approach is described based on the principle of centralized‐training and decentralized‐execution. Comparative control performances of MADQN‐BPB with a commercially available approach, and the global optimal solution are evaluated. Simulation results verify the capability of MADQN‐BPB in voltage stabilization with zero communication burden. Its control performance is close to that of the global optimal solution, and is scalable to the variations of environment.

Optimal robust online tracking control for space manipulator in task space using off-policy reinforcement learning

This study addresses the demands for adaptability, uncertainty management, and high performance in the control of space manipulators, and the inadequacies in achieving optimal control and handling external uncertainty in task space in previous research. Based on off-policy reinforcement learning, a model-free and time-efficient method for online robust tracking control in task space is devised. To address the complexity of dynamic equations in task space, a mixed-variable approach is adopted to transform the multivariable coupled time-varying problem into a single-variable problem. Subsequently, the optimal control policy is derived with the disturbance convergence, stability, and optimality of the control method being demonstrated. This marks the first instance of achieving robust optimal tracking control in task space for space manipulators. The efficacy and superiority of the presented algorithm are validated through simulation.

Wall Modeling of Turbulent Flows with Varying Pressure Gradients Using Multi-Agent Reinforcement Learning

We propose a framework for developing wall models for large-eddy simulation that is able to capture pressure-gradient effects using multi-agent reinforcement learning. Within this framework, the distributed reinforcement learning agents receive off-wall environmental states, including pressure gradient and turbulence strain rate, ensuring adaptability to a wide range of flows characterized by pressure-gradient effects and separations. Based on these states, the agents determine an action to adjust the wall eddy viscosity and, consequently, the wall-shear stress. The model training is in situ with wall-modeled large-eddy simulation grid resolutions and does not rely on the instantaneous velocity fields from high-fidelity simulations. Throughout the training, the agents compute rewards from the relative error in the estimated wall-shear stress, which allows them to refine an optimal control policy that minimizes prediction errors. Employing this framework, wall models are trained for two distinct subgrid-scale models using low-Reynolds-number flow over periodic hills. These models are validated through simulations of flows over periodic hills at higher Reynolds numbers and flows over the Boeing Gaussian bump. The developed wall models successfully capture the acceleration and deceleration of wall-bounded turbulent flows under pressure gradients and outperform the equilibrium wall model in predicting skin friction.

Optimal drift rate control and two-sided impulse control for a Brownian system with the long-run average criterion

Abstract In this paper, we consider a joint drift rate control and two-sided impulse control problem in which the system manager adjusts the drift rate as well as the instantaneous relocation for a Brownian motion, with the objective of minimizing the total average state-related cost and control cost. The system state can be negative. Assuming that instantaneous upward and downward relocations take a different cost structure, which consists of both a setup cost and a variable cost, we prove that the optimal control policy takes an $\left\{ {\!\left( {{s^{\ast}},{q^{\ast}},{Q^{\ast}},{S^{\ast}}} \right),\!\left\{ {{\mu ^{\ast}}(x)\,:\,x \in [ {{s^{\ast}},{S^{\ast}}}]} \right\}} \right\}$ form. Specifically, the optimal impulse control policy is characterized by a quadruple $\left( {{s^{\ast}},{q^{\ast}},{Q^{\ast}},{S^{\ast}}} \right)$ , under which the system state will be immediately relocated upwardly to ${q^{\ast}}$ once it drops to ${s^{\ast}}$ and be immediately relocated downwardly to ${Q^{\ast}}$ once it rises to ${S^{\ast}}$ ; the optimal drift rate control policy will depend solely on the current system state, which is characterized by a function ${\mu ^{\ast}}\!\left( \cdot \right)$ for the system state staying in $[ {{s^{\ast}},{S^{\ast}}}]$ . By analyzing an associated free boundary problem consisting of an ordinary differential equation and several free boundary conditions, we obtain these optimal policy parameters and show the optimality of the proposed policy using a lower-bound approach. Finally, we investigate the effect of the system parameters on the optimal policy parameters as well as the optimal system’s long-run average cost numerically.

Inventory rationing, admission control, and production capacity allocation in a make‐to‐stock/make‐to‐order manufacturing system

AbstractThis paper considers a manufacturing system in which products are produced in both make‐to‐stock (MTS) and make‐to‐order (MTO) modes. Production of MTS and MTO products is done in batches, incurs a setup cost, and is non‐preemptive. The inventory of MTS products fulfills the demand of multiple classes, and each class demand can be satisfied or rejected. Customer orders for MTO production can be accepted or rejected, and their size is the same as the production batch. The primary goal of this paper is to study a policy that coordinates inventory rationing, admission control, and production capacity allocation to maximize the system's profit. We formulate the problem as a Markov decision process model and identify the structure of optimal control policies. We investigate the effect of inventory rationing on the profit by comparing its performance to that of the system with a first‐come‐ first‐serve policy to allocate inventory to multiple demand classes and study the extent to which the benefit of inventory rationing can be affected by system parameter changes. We also propose a heuristic that manages control decisions from linear threshold functions. Our test results from numerical examples show that the average percentage difference between the optimal and heuristic policies is within 1.2%.

Delayed reinforcement learning converges to intermittent control for human quiet stance

The neural control of human quiet stance remains controversial, with classic views suggesting a limited role of the brain and recent findings conversely indicating direct cortical control of muscles during upright posture. Conceptual neural feedback control models have been proposed and tested against experimental evidence. The most renowned model is the continuous impedance control model. However, when time delays are included in this model to simulate neural transmission, the continuous controller becomes unstable. Another model, the intermittent control model, assumes that the central nervous system (CNS) activates muscles intermittently, and not continuously, to counteract gravitational torque. In this study, a delayed reinforcement learning algorithm was developed to seek optimal control policy to balance a one-segment inverted pendulum model representing the human body. According to this approach, there was no a-priori strategy imposed on the controller but rather the optimal strategy emerged from the reward-based learning. The simulation results indicated that the optimal neural controller exhibits intermittent, and not continuous, characteristics, in agreement with the possibility that the CNS intermittently provides neural feedback torque to maintain an upright posture.

Asymptotic Scaling of Optimal Cost and Asymptotic Optimality of Base-Stock Policy in Several Multidimensional Inventory Systems

Asymptotic Optimality of Simple Heuristic Policies for Multidimensional Inventory Systems Stochastic inventory systems with multidimensional state spaces, such as lost-sales system with positive lead time and perishable inventory system, are challenging to manage because of the curse of dimensionality, and their optimal control policies are extremely complex. In “Asymptotic Scaling of Optimal Cost and Asymptotic Optimality of Base-Stock Policy in Several Multidimensional Inventory Systems,” Bu, Gong and Chao consider three classes of such systems in the regime of large unit penalty cost, and they establish the asymptotic optimality of (modified) simple base-stock policies as well as an explicit expression for the optimal cost rate in each of these systems. These results justify the applications of such policies in real-world applications, and they are achieved by constructing tight newsvendor upper and lower bounds on the systems’ costs and analyzing the asymptotic scaling of newsvendor costs with large unit penalty cost. This approach is expected to be useful in studying other multidimensional stochastic inventory systems.