Near-optimal Control Research Articles

Iterative Online Optimal Feedback Control

This article proposes a data-driven iterative feedback control method to efficiently solve finite time horizon, nonlinear, input constrained optimal control problems. The proposed method introduces a novel approach to combine an inexact system model with measured state information to reduce the cost and provide near-optimal control by approximately solving the optimal control problem along the trajectory of the real system, as opposed to solving it along the trajectory predicted by the inexact model. We present a new algorithm that implements the proposed method, establish the convergence and optimality properties of the proposed algorithm, and compare it to optimal feedback control and model predictive control that solve the same optimal control problem along the trajectory predicted by the inexact model. Finally, we illustrate the generality of the proposed algorithm by approximately solving a challenging optimal control problem with unknown and changing dynamics.

CEFL: Online Admission Control, Data Scheduling, and Accuracy Tuning for Cost-Efficient Federated Learning Across Edge Nodes

With the proliferation of Internet of Things (IoT), zillions of bytes of data are generated at the network edge, incurring an urgent need to push the frontiers of artificial intelligence (AI) to network edge so as to fully unleash the potential of the IoT big data. To materialize such a vision which is known as edge intelligence, federated learning is emerging as a promising solution to enable edge nodes to collaboratively learn a shared model in a privacy-preserving and communication-efficient manner, by keeping the data at the edge nodes. While pilot efforts on federated learning have mostly focused on reducing the communication overhead, the computation efficiency of those resource-constrained edge nodes has been largely overlooked. To bridge this gap, in this article, we investigate how to coordinate the edge and the cloud to optimize the system-wide cost efficiency of federated learning. Leveraging the Lyapunov optimization theory, we design and analyze a cost-efficient optimization framework CEFL to make online yet near-optimal control decisions on admission control, load balancing, data scheduling, and accuracy tuning for the dynamically arrived training data samples, reducing both computation and communication cost. In particular, our control framework CEFL can be flexibly extended to incorporate various design choices and practical requirements of federated learning, such as exploiting the cheaper cloud resource for model training with better cost efficiency yet still facilitating on-demand privacy preservation. Via both rigorous theoretical analysis and extensive trace-driven evaluations, we verify the cost efficiency of our proposed CEFL framework.

Event-triggered receding horizon control via actor-critic design

In this paper, we propose a novel event-triggered near-optimal control for nonlinear continuous-time systems. The receding horizon principle is utilized to improve the system robustness and obtain better dynamic control performance. In the proposed structure, we first decompose the infinite horizon optimal control into a series of finite horizon optimal problems. Then a learning strategy is adopted, in which an actor network is employed to approximate the cost function and an critic network is used to learn the optimal control law in each finite horizon. Furthermore, in order to reduce the computational cost and transmission cost, an event-triggered strategy is applied. We design an adaptive trigger condition, so that the signal transmissions and controller updates are conducted in an aperiodic way. Detailed stability analysis shows that the nonlinear system with the developed event-triggered optimal control policy is asymptotically stable. Simulation results on a single-link robot arm with different noise types have demonstrated the effectiveness of the proposed method.

Near-optimal neural-network robot control with adaptive gravity compensation

Adaptive nonlinear optimal control methods, as proposed in the literature, give rise to some questions around practical implementation in robotics, especially how to find a solution in a reasonable time and how to deal with gravity. This paper proposes a method to solve these problems by using a neural network with local basis-function domains, specifically the Cerebellar Model Articulation Controller (CMAC). The algorithm uses the local domains in order to keep track of the value of local cost-functionals. In this way, it can freeze the learning of the network’s weights in a feedforward-like component in the CMAC when the bias has been overcome identified by using an error-based cost-functional e.g. automatic gravity compensation in a robot. After the feedforward component has been established, the algorithm then starts to learn another set of weights which contribute to feedback-like terms in the CMAC and these weights get frozen when they no longer reduce a cost-functional that includes additional control effort e.g. in a robot the control effort beyond that needed to compensate for gravity is penalized. Lyapunov methods guarantee uniformly ultimately bounded signals and ensure weight drift and bursting do not occur. One advantage is that the training time for finding a near-optimal control does not increase over previous neural-adaptive methods. Another advantage is that penalizing the control effort in a search for optimization does result in any steady-state error due to gravity. Simulations show that the proposed method significantly outperforms a standard adaptive-CMAC control using e-modification, without increasing control effort or training time. An experimental flexible-joint robot verifies that the adaptive method significantly outperforms a linear quadratic regulator.

Event-triggered constrained control with DHP implementation for nonaffine discrete-time systems

This paper proposes an event-based near-optimal control algorithm for nonaffine discrete-time systems with constrained inputs. The method is derived from the dual heuristic dynamic programming (DHP) technique. The challenge caused by saturating actuators is overcome by using a nonquadratic performance index. Then, the event-based control technique is used to decrease the amount of computation. Meanwhile, the stability analysis is provided. It illustrates that the proposed event-based method can asymptotically stabilize the nonaffine systems by using the Lyapunov method. Furthermore, the stability conditions and the design process of the event-based controller are established. The event-based DHP algorithm is implemented by constructing three neural networks, namely, the model network, the critic network, and the action network. Finally, simulation studies are conducted to demonstrate the applicability and the performance of the proposed method.

Near-optimal control problems for forward-backward regime-switching systems

This paper investigates the near-optimality for a class of forward-backward stochastic differential equations (FBSDEs) with continuous-time finite state Markov chains. The control domains are not necessarily convex and the control variables do not enter forward diffusion term. Some new estimates for state and adjoint processes arise naturally when we consider the near-optimal control problem in the framework of regime-switching. Inspired by Ekeland’s variational principle and a spike variational technique, the necessary conditions are derived, which imply the near-minimum condition of the Hamiltonian function in an integral sense. Meanwhile, some certain convexity conditions and the near-minimum condition are sufficient for the near-optimal controls with order ε1/2. A recursive utility investment consumption problem is discussed to illustrate the effectiveness of our theoretical results.

Data-Driven Nonlinear Near-Optimal Regulation Based on Multi-Dimensional Taylor Network Dynamic Programming

Using the data-driven control formulation, an iterative dynamic programming approach which is based on a multi-dimensional Taylor network is established to design the near optimal regulation of discrete-time nonlinear systems. For discrete-time general nonlinear systems, the iterative adaptive dynamic programming algorithm is developed and proved to guarantee the property of convergence and optimality. Three networks are constructed, namely, the identification network, critic network and action network. Moreover, a globalized dual heuristic programming technique with detailed implementation is developed. The cost function and its derivative can be approximated by this novel architecture. Besides, without the consideration of the system dynamics, this technique can learn the near-optimal control law simultaneously and adaptively. In addition, this technique greatly improves the existing results of the iterative adaptive dynamic programming algorithm in terms of reducing the requirement of the control matrix. Furthermore, because of the approach that is based on the multi-dimensional Taylor network, the amount of calculation needed is also greatly reduced. The simulation experiment is described to illustrate the effectiveness of the data-driven optimal regulation method proposed in this paper.

Data-based reinforcement learning for lane keeping with input saturation

With the development of artificial intelligence, autonomous driving has received extensive attention. As a very complex integrated system, the autonomous vehicle has several modules. This paper is related to the control module, which is used to design an optimal or near-optimal controller to control the desired trajectory of the vehicle. In this paper, lateral control strategy for lane keeping task is proposed based on the model-free reinforcement learning. Different from the model-based methods such as linear quadratic regulator and model predictive control, our method only requires the generated data rather than the perfect knowledge of the system model to guarantee the optimal performance. At the same time, in order to meet two needs of passengers' comfort and fuel economy, input saturation should be considered in the design of the control module. A low-gain state feedback control method is adopted. It mainly solves some algebraic Riccati equations for data-based lateral control. Finally, the corresponding simulation is given and the validity of the algorithm is verified.

Continuous Near-Optimal Control of Energy Storage Systems

Energy storage plays an essential role in enabling greater uptake of renewable generation. In many applications, energy storage must be used for multiple (sometimes competing) purposes in order to provide the maximum possible economic return. A common approach is to find an optimal sequence of charge and discharge rates for a set of discrete time intervals across the horizon of interest. However, calculating optimal solutions at a high temporal resolution can be computationally expensive. In addition, conditions can change instantly as renewable generation fluctuates and loads are switched on and off. What is needed, therefore, is an approach that can both track some long-term optimal trajectory, while still responding to changing conditions in real-time. In this work, we propose two approaches that aim to achieve this. In the first, we calculate a discrete optimal solution, but then convert this into a schedule for simple rule-based controllers that can respond to changes continuously. In the second, we use historical optimal solutions and rule-based schedules to train a neural network that generates a similar schedule. We find that both approaches offer significant advantages over standard discrete optimal solutions: they provide a similar amount of value (and in some cases more), while being 30x less computationally expensive to compute.

A Digital Receding-Horizon Learning Controller for Nonlinear Continuous-time Systems

Abstract The integration of reinforcement learning (RL) and model predictive control (MPC) is promising for solving nonlinear optimization problems in an efficient manner. In this paper, a digital receding horizon learning controller is proposed for continuous-time nonlinear systems with control constraints. The main idea is to develop a digital design for RL with actor-critic design (ACD) in the framework of MPC, to realize near-optimal control of continuous-time nonlinear systems. Different from classic RL for continuous-time systems, the actor adopted is learned in discrete-time steps, while the critic evaluates the learned control policy continuously in the time domain. Moreover, we use soft barrier functions to deal with control constraints and the robustness of the actor-critic network is proven. A simulation example is considered to show the effectiveness of the proposed approach.

The connection between discrete and continuous state constrained optimal control systems

An optimal control problem driven by a one-dimensional ordinary differential equation under continuous state constraints is considered in this study. From an operational point of view, we introduce a discrete state constrained optimal control problem and prove that this problem is a near-optimal control problem of the continuous state constraints problem. Furthermore, we show that the optimal solution of the near-optimal control problem converges to the optimal solution of the continuous state constraints problem as the number of the discrete set approaches to infinity. Finally, we use a linear-quadratic optimal problem to verify the main results of this study.

Generalized Optimal and Explicit PI/PID Tuning Formulas for Underdamped Second-order Systems

This study aims to propose generalized formulas in an explicit form to optimally tune the gains of the proportional-integral (PI) and proportional-integral-derivative (PID) controllers of an underdamped second-order system. Unlike other methods used in literature, the controller gains are tuned explicitly based on the known system parameters of the second-order system, namely: the damping ratio and the natural frequency. The proposed accurate and one-step formulas do not need any further effort to effectively control any given second-order system while keeping the desired performance indices within near-optimal specifications. The proposed work relies on a restricted deadbeat response criterion and cascaded surface curve fitting procedure to end by near-optimal controller gains tuning formulas. A particle swarm optimization (PSO) method is employed to optimize the primary tuned gains for wide-range of system frequency. The proposed formulas are tested and simulated to demonstrate their effectiveness with comparisons with the closest common related gains tuning methods. Based on the comprehensive analysis, the proposed work ensures 2% maximum percentage overshoot when the proposed formulas are adopted to tune the gains of the PID controllers. Different comparisons are carried out to show the effectiveness of the proposed work.

Order reduction of a wind turbine energy system via the methods of system balancing and singular perturbations

In this paper we study the order reduction of a wind turbine model using the methods of balancing transformation and singular perturbations. We show that the order of the considered wind turbine model can be reduced from eight to six via the balancing transformation. Further reduction via the aforementioned method results in a significant jump in the error bound. In contrast, the method of singular perturbations shows that the order of the model can be further reduced to four, or two, and still provide very good approximations to the system model, in terms of its transient step response. Moreover, we show that the reduction in model order achieved via singular perturbations is superior to that achieved via balancing, when the linear-quadratic near-optimal controllers are considered and when wind turbulence and a large-signal disturbance are applied to the system.

Event-Triggered Adaptive Dynamic Programming for Non-Zero-Sum Games of Unknown Nonlinear Systems via Generalized Fuzzy Hyperbolic Models

In this paper, by incorporating the event-triggered mechanism and the adaptive dynamic programming algorithm, a novel near-optimal control scheme for a class of unknown nonlinear continuous-time non-zero-sum (NZS) differential games is investigated. First, a generalized fuzzy hyperbolic model based identifier is established, using only the input-output data, to relax the requirement for the complete system dynamics. Then, under the event-based framework, the coupled Hamilton-Jacobi equations are derived for the multiplayer NZS games. Then, the adaptive critic design method is employed to approximate the optimal control policies; thus, an identifier-critic architecture is developed to obtain the event-triggered controller. By the virtue of the Lyapunov theory, a state-dependent triggering condition, which is different from the existing works, is developed to achieve the stability of the closed-loop control system both for the continuous and jump dynamics. Finally, two numerical examples are simulated to substantiate the feasibility of the analytical design.

Sufficient and necessary conditions for stochastic near-optimal controls: A stochastic chemostat model with non-zero cost inhibiting

Near-optimal controls are as important as optimal controls for both theory and applications. Meanwhile, using inhibitor to control harmful microorganisms and ensure maximum growth of beneficial microorganisms (target microorganisms) is a very interesting topic in the chemostat. Thus, in this paper, we consider a stochastic chemostat model with non-zero cost inhibiting in finite time. The near-optimal control problem was constructed by minimizing the number of harmful microorganisms and minimizing the cost of inhibitor. We find that the Hamiltonian function is key to estimate objective function, and according to the adjoint equation, we obtain some error estimations of the near-optimality. Finally, we establish sufficient and necessary conditions for stochastic near-optimal controls of this model and numerical simulations and some conclusions are given.

Optimal vaccination strategy for an SIRS model with imprecise parameters and Lévy noise

Parameters of mathematical models are often imprecise due to various uncertainties. How parameter imprecision and sudden environmental changes influence the optimal control of dynamical systems remains unclear. In this paper, we formulate an Susceptible-Infected-Recovered-Susceptible (SIRS) epidemic model that includes imprecise parameters, Lévy jumps, and vaccination control. We use the model to investigate the near-optimal control problem in the setting of vaccination. We obtain priori estimates of the susceptible, infected and recovered populations. We establish sufficient and necessary conditions for the near-optimality of the model using Pontryagin stochastic maximum principle. We also develop an algorithm for the near-optimal control problem and perform numerical simulations to illustrate the effect of vaccination and Lévy noise.

A case for the adoption of decentralised reinforcement learning for the control of traffic flow on South African highways

As an alternative to capacity expansion, various dynamic highway traffic control measures have been introduced. Ramp metering and variable speed limits are often considered to be effective dynamic highway control measures. Typically, these control measures have been employed in conjunction with either optimal control methods or online feedback control. One shortcoming of feedback control is that it provides no guarantee of optimality with respect to the chosen metering rate or speed limit. Optimal control approaches, on the other hand, are limited in respect of their applicability to large traffic networks due to their significant computational expense. Reinforcement learning is an alternative solution approach, in which an agent learns a near-optimal control strategy in an online manner, with a smaller computational overhead than those of optimal control approaches. In this paper an empirical case is made for the adoption of a decentralised reinforcement learning approach towards solving the control problems posed by both ramp metering and variable speed limits simultaneously, and in an online manner. The effectiveness of this approach is evaluated in the context of a microscopic traffic simulation model of a section of the N1 national highway outbound from Cape Town in South Africa's Western Cape Province.

Dynamic Near-Optimal Control Allocation for Spacecraft Attitude Control Using a Hybrid Configuration of Actuators

This paper proposes a novel dynamic near-optimal control allocation scheme with combination of a saturated baseline controller for spacecraft attitude control using single-gimbal control moment gyros (CMGs) and reaction wheels. First, a saturated controller is proposed to stabilize the nominal system in the presence of actuation mismatch. Aided by a state-dependent variable, a dynamic control allocator is then proposed that allows for smooth switching between two actuation sets. Unlike the previous static constraint optimization formulations, the control allocation augments its performance function with penalty terms in order to enforce individual input constraints and configuration singularity avoidance. Moreover, this dynamic control allocation is implemented with an online update law, which has a modest computational complexity compared to its numerical optimization counterpart. The closed-loop boundedness is guaranteed by a constructive Lyapunov-design method. Simulation results demonstrate that during the attitude maneuvers, the input saturation constraint and the active avoidance of CMG singularities are enforced with a relatively small computational cost.

Deep Neural Networks Based Real-time Optimal Control for Lunar Landing

Recent research on deep learning control, a new control algorithm based on machine learning able to learn deep architectures, has shown excellent performance on robots and drones. With the development of intelligent control like deep learning and reinforcement learning, accuracy, real-time, adaptability, robustness and autonomy of control algorithm have been achieved by the intelligent controls. Traditional control methods have difficulties to achieve nice performance in complex situations. Deep learning offers powerful algorithms to real-time search near-optimal controllers of lunar landing spacecraft with nonlinear dynamics. In terms of lunar landing control system, deep architectures offer the possibility to get an approximate solution of co-state equation without time-consuming iterative process. Furthermore, real-time optimal thrust during lunar landing may be derived directly through deep neural networks. As a single infrastructure for machine learning in both production and research, TensorFlow is chosen for training the deep artificial neural networks in this paper. Numerical simulations demonstrate the effectiveness of deep neural networks. The results of deep neural networks based optimal control are contrasted with traditional optimal algorithm, whose main idea is to track the pre-designed optimal trajectory by ground station. This research provides an effective approach to cope with the lunar landing problem.

Fixed-time near-optimal control for repointing maneuvers of a spacecraft with nonlinear terminal constraints

Repointing maneuvers of a spacecraft in staring mode are investigated where the optical axis is required to align with the target orientation. Different from traditional three-axis reorientation maneuvers, the rotation about the optical axis is free of constraints for repointing maneuvers. Both static target observation and moving target detection constraints are considered. The problem is then formulated as a finite-time horizon optimal control problem with nonlinear terminal constraints. A simple and efficient state-dependent Riccati equation(SDRE) based dynamic programming approach is applied to tackle this nonlinear optimal control problem. The convergence of the attitude from initial conditions to the desired terminal constraint is rigorously proved for the first time. Considering the inability of the SDRE method to deal with the problem of large angle maneuvers, an improved SDRE approach combined with a waypoint is proposed to enhance control performance. Finally, numerical investigations are conducted and compared with the real optimal solutions obtained by using the optimization software.