Near-optimal Control Strategy Research Articles

Performance-prescribed optimal neural control for hypersonic vehicles considering disturbances: An adaptive dynamic programming approach

The precise trajectory tracking control of hypersonic vehicles plays a crucial role in secure flight. To address this challenge, this article develops an optimal neural control framework with prescribed performance constraints for height-velocity control tasks with disturbances. To this end, the control-oriented error model is firstly established and transformed into an affine nonlinear form to provide a foundation for the design of controller. Then, in order to balance performance constraints and convergence speed, a performance-constraint function is introduced to convert the constrained tracking error into an unconstrained one. After that, an improved sliding mode compensation observer is proposed by combining low-fidelity approximation data with high-fidelity compensation model for eliminating disturbances. The developed control scheme is organized by an adaptive dynamic programming (ADP) algorithm based on the actor-critic policy structure to achieve the near-optimal control strategy. Meanwhile, the designed neural network (NN) weight updating laws are provided to obtain the solution of Hamiltonian-Jacobi-Bellman (HJB) equation rapidly without high calculation consumption. Finally, the stability conditions of the closed-loop system are performed and ensure the entire framework is semi-global uniformly ultimately bounded (SGUUB), two simulation cases verify the effectiveness and superiority of the proposed approach.

Adaptive dynamic programming base on MMC device of a flexible high-altitude long endurance aircraft

Flexible high-altitude long endurance (HALE) aircraft face challenges such as the delay effect caused by aeroelasticity and low efficiency during high-altitude cruising. This paper addresses a new active control scheme that replaces traditional aerodynamic control surfaces with longitudinal and lateral moving masses and adopts an observer-based adaptive dynamic programming (ADP) control strategy, aiming to solve the attitude control of HALE aircraft with flexible wings. Firstly, considering the account of unsteady aerodynamics, attitude angular velocity, and moving masses, complete dynamic model is established for a flexible aircraft, in order to more accurately describe the state of the HALE aircraft. This model includes two moving masses to replace traditional ailerons and elevators. It is difficult to design attitude control strategies considering the delay effects and unknown dynamic disturbances of HALE aircraft. To this end, the actor-critic structure of the ADP approach is designed to achieve the near-optimal control strategy of the system, which eliminates the need for prior knowledge of model parameters. A fixed time disturbance observer (FTDO) is proposed to estimate the future tracking error information and unknown disturbances to improve the control performance of the attitude. Finally, the stability of the system is proved via the Lyapunov technique and a comprehensive simulation of HALE aircraft is provided. The simulation results show that the proposed control algorithm can enable the aircraft attitude angles to quickly response and maintain stability under gust wind. Compared with a single ADP control strategy, this paper's algorithm has advantages in response speed and error.

Vibration Control with Reinforcement Learning Based on Multi-Reward Lightweight Networks

This paper proposes a reinforcement learning method using a deep residual shrinkage network based on multi-reward priority experience playback for high-frequency and high-dimensional continuous vibration control. Firstly, we keep the underlying equipment unchanged and construct a vibration system simulator using FIR filters to ensure the complete fidelity of the physical model. Then, by interacting with the simulator using our proposed algorithm, we identify the optimal control strategy, which is directly applied to real-world scenarios in the form of a neural network. A multi-reward mechanism is proposed to assist the lightweight network to find a near-optimal control strategy, and a priority experience playback mechanism is used to prioritize the data to accelerate the convergence speed of the neural network and improve the data utilization efficiency. At the same time, the deep residual shrinkage network is introduced to realize adaptive denoising and lightweightness of the neural network. The experimental results indicate that under narrowband white-noise excitation ranging from 0 to 100 Hz, the DDPG algorithm achieved a vibration reduction effect of 12.728 dB, while our algorithm achieved a vibration reduction effect of 20.240 dB. Meanwhile, the network parameters were reduced by more than 7.5 times.

Reinforcement Learning for Dual-Control Aircraft Six-Degree-of-Freedom Attitude Control with System Uncertainty

This article proposes a near-optimal control strategy based on reinforcement learning, which is applied to the six-degree-of-freedom (6-DoF) attitude control of dual-control aircraft. In order to solve the problem that the existing reinforcement learning is difficult to apply to the high-dimensional multiple-input multiple-output (MIMO) systems, the Long Short-Term Memory (LSTM) neural network is introduced to replace the polynomial network in the adaptive dynamic programming (ADP) technique. Meanwhile, based on the Lyapunov method, a novel online adaptive updating law of LSTM neural network weights is given, and the stability of the system is verified. In the simulation process, the algorithm proposed in this article is applied to the six-degree-of-freedom attitude control problem of dual-control aircraft with system uncertainty. The simulation results show that the algorithm can achieve near-optimal control.

A linear-quadratic mean-field game of backward stochastic differential equation with partial information and common noise

This paper studies a linear-quadratic (LQ) mean-field game of stochastic large-population system with partial information and common noise, where the large-population system satisfies a class of backward stochastic differential equations (BSDEs), and both coupling structure and ki (a part of solution of BSDE) enter state equations and cost functionals. By virtue of stochastic maximum principle and optimal filter technique, we obtain a Hamiltonian system first, which is a fully-coupled forward-backward stochastic differential equation (FBSDE). Decoupling the Hamiltonian system, we obtain three ordinary differential equations (ODEs), a forward and a backward optimal filtering equations, which enable us to derive an optimal control of an auxiliary limiting control problem. Next, we verify that a decentralized control strategy is an ϵ-Nash equilibrium of the LQ game. Finally, we solve a product pricing problem and a network security control problem in applications. We give a near-optimal price strategy and a near-optimal control strategy with numerical simulations, respectively.

ADP-Based Attitude-Tracking Control with Prescribed Performance for Hypersonic Vehicles

This paper proposes an observer-based adaptive dynamic programming (ADP) approach to handle the optimal attitude-tracking control tasks for hypersonic vehicles with prescribed performance constraints. Herein, the biggest challenge is how to design an optimal attitude-tracking controller which could balance performance constraints and optimization capabilities with the presence of partially unknown dynamics and actuator saturation. To this end, the transformation function is designed to convert the constrained tracking error into the equivalent unconstrained one. After that, a constraint-aware neural network-based observer is proposed to estimate the future tracking error information required by the optimal controller, which avoids the requirement of the prior nonlinear dynamics knowledge, and its estimation errors are compensated through online feedback. Taking the estimated future attitude-tracking error into account, the critic-only ADP approach is designed via the value iteration to achieve the near-optimal control strategy for equivalent systems with a simplified optimized control structure. Furthermore, a novel experience replay-driven weight updating law is provided to circumvent the conventional persistent excitation condition with an online-verification condition, and the algorithm convergence and closed-loop stability are discussed with the help of the Lyapunov technique and induction. Finally, a comprehensive simulation of an attitude-tracking control system for hypersonic vehicles is provided to verify the effectiveness and superiority of the proposed approach.

Multi-objective control and energy management strategy based on deep Q-network for parallel hybrid electric vehicles

To promote the energy management strategy of hybrid electric vehicles towards the direction of intelligence, this paper proposes a control model to design the learning-based energy management strategy for a parallel hybrid electric vehicle, which uses the deep Q-network (DQN) algorithm of deep reinforcement learning to control the engine and continuously variable transmission (CVT) synchronously. After completing the offline training of the control model, the near-optimal control strategy fitted by the neural network parameters is saved, and it is loaded and tested directly during the online test, which can reflect if the neural network has learned the mapping relationship between a random state and the optimal action. The simulation result of the online test shows that the DQN-based EMS can achieve a fuel economy of 5.31L/100km, and the consuming time is 1.67s when running the testing cycle of 1686s, which can ensure the real-time application potential, adaptability, and robustness.

Synchronous Fault-Tolerant Near-Optimal Control for Discrete-Time Nonlinear PE Game.

In this article, the synchronous fault-tolerant near-optimal control strategy design problem is studied for a class of discrete-time nonlinear pursuit-evasion (PE) games. In the studied PE game, the input saturation phenomenon and possible actuator fault are simultaneously taken into consideration. To accelerate the estimation speed, a novel nonlinear fault estimator is designed by introducing a nonlinear function. Then, for the purpose of obtaining the synchronous control strategy for the discrete-time PE games, an approximate Hamilton-Jacobi-Isaacs (HJI) equation is established, which is seldom utilized for the discrete-time approximate dynamic programming in most existing results. It should be noticed that the synchronous control strategy designed based on the approximate HJI equation can be convergent very fast because of its quasi-Newton's iteration form. Furthermore, the sufficient condition is established to guarantee that the studied system is uniformly ultimately bounded. Finally, a numerical simulation of the hypersonic vehicle system is carried out to validate the proposed methodology.

A case for the adoption of decentralised reinforcement learning for the control of traffic flow on South African highways

As an alternative to capacity expansion, various dynamic highway traffic control measures have been introduced. Ramp metering and variable speed limits are often considered to be effective dynamic highway control measures. Typically, these control measures have been employed in conjunction with either optimal control methods or online feedback control. One shortcoming of feedback control is that it provides no guarantee of optimality with respect to the chosen metering rate or speed limit. Optimal control approaches, on the other hand, are limited in respect of their applicability to large traffic networks due to their significant computational expense. Reinforcement learning is an alternative solution approach, in which an agent learns a near-optimal control strategy in an online manner, with a smaller computational overhead than those of optimal control approaches. In this paper an empirical case is made for the adoption of a decentralised reinforcement learning approach towards solving the control problems posed by both ramp metering and variable speed limits simultaneously, and in an online manner. The effectiveness of this approach is evaluated in the context of a microscopic traffic simulation model of a section of the N1 national highway outbound from Cape Town in South Africa's Western Cape Province.

Near-Optimal Control for Time-Varying Linear Discrete Systems With Additive Nonlinearities and Random Gains

In this technical note, the near-optimal control problem is investigated for a class of time-varying linear discrete systems subject to additive nonlinearities and random gains. First, certain upper bound on the cost function is proposed to quantify the control performance. Subsequently, in virtue of the time-varying discrete Riccati-like recursions (DRLRs), the near-optimal control strategy is designed such that this upper bound is minimized over the finite horizon. To reflect the steady-state behavior of the dynamic system, the upper bound of the derived time-varying DRLRs is also analyzed over the infinite horizon by resorting to the semidefinite programming method. Finally, a numerical example is provided to verify the validity and advantages of the proposed methodology.

Integration of On-Line Control in Optimal Design of Multimode Power-Split Hybrid Electric Vehicle Powertrains

The multimode power-split architecture for hybrid electric vehicle (HEV) powertrains is generally known for the complexity of its operation. This paper first addresses the challenge of developing an automated on-line near-optimal control strategy for these systems. Particularly, a machine learning logic based on supervised learning is developed for on-line selection of the HEV operating mode, thus the particular set of clutches to be engaged. An efficiency-based approach is then adopted to determine the optimal power-split between powertrain components. Later, the developed strategy finds integration in an optimal design methodology for multimode power-split HEVs considering the effectiveness and ease of on-line controllability. The obtained results are compared with the ones by the traditional HEV design methodology that only considers off-line energy management. The illustrated design methodology with on-line control reveals efficient at identifying the multimode HEV design that demonstrates optimal predicted fuel economy values and ease of on-line controllability simultaneously. Results suggest that the HEV design optimization procedure may produce different outcomes and demonstrate more effective when the evaluation of the on-line controlled operation is integrated.

Benchmark fuel economy for a parallel hybrid electric three-wheeler vehicle (rickshaw)

The core contribution to this work is the development of benchmark fuel economy for a three-wheeler hybrid electric rickshaw and its comparison with heuristics controllers designed with optimal and non-optimal rules. Dynamic programming is used as a feasible technique for powertrain benchmark analysis. A parallel hybrid electric three-wheeler vehicle is modeled in MATLAB/Simulink through forward facing simulator. The dynamic programming technique is employed through the backward facing simulator, ensuring optimal power sharing between two energy sources (engine and motor) while keeping the battery state of charge in the charge-sustaining mode. The extracted rules from dynamic programming forming near-optimal control strategies are playing a vital role in deciding overall fuel consumption. Unlike the dynamic programming control actions, these extracted rules are implementable through the forward facing simulator. From the simulation results, it can be concluded that a substantial improvement of fuel economy is achieved through the application of dynamic programming. Rule-based (near-optimal) strategy using dynamic programming results shows about 9% more fuel consumption as compared with the dynamic programming (benchmark solution), which is then compared with non-optimal rule-based heuristics controller. It is shown that non-optimal rule-based controller has 18% more fuel consumption than dynamic programming results.

Optimal and near-optimal indoor temperature and humidity controls for direct load control and proactive building demand response towards smart grids

Abstract Shutting down part of operating chillers directly in central air-conditioning systems of buildings to meet the urgent demand reduction needs of power grids has received increasing attention recently. However, due to limited cooling supply during above demand response (DR) events, the indoor air temperature and particularly relative humidity would often increase to unacceptable levels, resulting in the failures of DR controls. Considering the restriction on power use during DR events, rational use of limited cooling supply turns out to be the inevitable choice. The feedback control strategies commonly-used today cannot properly deal with the environment and system control issues under limited cooling supply during DR events. However, no study on this problem can be found in the research literature. As the first effort, two control strategies (i.e., optimal and near-optimal) are developed to address the environment control issues (concerning both indoor temperature and humidity controls) under a pre-determined power limiting threshold during DR events. The optimal control strategy optimizes the air flow set-points of individual AHUs (air handling units) using model-based prediction and genetic algorithm to achieve the best possible indoor environment control. The near-optimal control strategy approaches such best environment control using a simple empirical method. Case studies are conducted and the results show that the air flow settings have significant impacts on the indoor environment controlled under limited cooling supply. Both control strategies can achieve significant improvements in the indoor temperature and humidity controls as well as significant fan power saving.

LQR Based Battery Charge Sustaining Strategy for Hybrid Electric Vehicle

Rule-based near-optimal control strategy derived from global optimization results has the best real-time performance among current online optimal control strategies of the hybrid electric vehicle. However, the battery charge sustainability based on the rule-based near-optimal control strategy is not theoretically studied in current researches, which is important for robust control in complex driving conditions. In this paper, a deviation correction strategy based on linear quadratic regulator (LQR) is proposed for a power-split hybrid electric bus in order to sustain the battery’s state of charge (SOC). The LQR controller is designed according to an equivalent circuit model based on the state feedback control. Offline simulation is carried out to verify the proposed strategy under different driving cycles. Results show that, with the proposed strategy, the SOC can converge to a predefined value at the end of the optimization horizon.

A new method for optimal control of a dual-mode CO2 heat pump with thermal storage

A dual-mode CO2 heat pump coupled with hot and cold thermal storages was proposed as a high-efficiency smart grid enabling option for supporting higher penetration levels of intermittent renewables in the energy system. However, the optimal control of simultaneous hot and cold thermal storage systems in real time is very delicate. In this study, a novel system-based optimization method is proposed, in which the COP of such system is simplified with empirical correlation, and a solution for the optimal control vector that maximize the transient system COP is determined using genetic algorithm for global optimization. To evaluate the system-based optimization method, experimental tests were conducted, and a real-time near-optimal control strategy for charging of dual-model energy storage systems was developed and implemented. Practical operation results show that the near-optimal control strategy is able to search the near-optimal transient total COP in real time, and optimize the overall COP of such dual-mode coupled systems during energy charging.

Energy Near-Optimal Control Strategies for Industrial and Traction Drives with a.c. Motors

The main contribution of this paper is a new rest-to-rest position control system for use with electric drives employing a.c. motors that is near-optimal with respect to combined electrical and frictional energy waste minimization. The friction has constant, linear, and quadratic components with respect to the rotor speed. The closeness to optimality is assessed by simulation, comparing the energy loss of the new control system with that predicted by computed optimal controls. The application of the near-optimal control system is rendered straightforward by using a symmetrical trapezoidal speed-time profile. This is provided by an energy saving reference position generator whose output is faithfully followed by means of a feedback control law based on forced dynamics control yielding prescribed closed loop dynamics, together with a matched zero dynamic lag precompensator. For load torque consisting of constant, linear, and quadratic components also maneuver time is optimized if it can be chosen arbitrary. Two case studies, one applied to position control of rotational drive and second one applied to train movement, confirm the possibilities of achieving energy savings.

Development of Near Optimal Rule-Based Control for Plug-In Hybrid Electric Vehicles Taking into Account Drivetrain Component Losses

A near-optimal rule-based mode control (RBC) strategy was proposed for a target plug-in hybrid electric vehicle (PHEV) taking into account the drivetrain losses. Individual loss models were developed for drivetrain components including the gears, planetary gear (PG), bearings, and oil pump, based on experimental data and mathematical governing equations. Also, a loss model for the power electronic system was constructed, including loss from the motor-generator while rotating in the unloaded state. To evaluate the effect of the drivetrain losses on the operating mode control strategy, backward simulations were performed using dynamic programming (DP). DP selects the operating mode, which provides the highest efficiency for given driving conditions. It was found that the operating mode selection changes when drivetrain losses are included, depending on driving conditions. An operating mode schedule was developed with respect to the wheel power and vehicle speed, and based on the operating mode schedule, a RBC was obtained, which can be implemented in an on-line application. To evaluate the performance of the RBC, a forward simulator was constructed for the target PHEV. The simulation results show near-optimal performance of the RBC compared with dynamic-programming-based mode control in terms of the mode operation time and fuel economy. The RBC developed with drivetrain losses taken into account showed a 4%–5% improvement of the fuel economy over a similar RBC, which neglected the drivetrain losses.

Dynamical Allocation of Cellular Resources as an Optimal Control Problem: Novel Insights into Microbial Growth Strategies.

Microbial physiology exhibits growth laws that relate the macromolecular composition of the cell to the growth rate. Recent work has shown that these empirical regularities can be derived from coarse-grained models of resource allocation. While these studies focus on steady-state growth, such conditions are rarely found in natural habitats, where microorganisms are continually challenged by environmental fluctuations. The aim of this paper is to extend the study of microbial growth strategies to dynamical environments, using a self-replicator model. We formulate dynamical growth maximization as an optimal control problem that can be solved using Pontryagin’s Maximum Principle. We compare this theoretical gold standard with different possible implementations of growth control in bacterial cells. We find that simple control strategies enabling growth-rate maximization at steady state are suboptimal for transitions from one growth regime to another, for example when shifting bacterial cells to a medium supporting a higher growth rate. A near-optimal control strategy in dynamical conditions is shown to require information on several, rather than a single physiological variable. Interestingly, this strategy has structural analogies with the regulation of ribosomal protein synthesis by ppGpp in the enterobacterium Escherichia coli. It involves sensing a mismatch between precursor and ribosome concentrations, as well as the adjustment of ribosome synthesis in a switch-like manner. Our results show how the capability of regulatory systems to integrate information about several physiological variables is critical for optimizing growth in a changing environment.

Comparison of Fuel Consumption and Fuel Cell Degradation Using an Optimised Controller

The Energy Management Strategy (EMS) of any hybrid vehicle is responsible for determining the operating state of many components on board the vehicle and therefore has significant effect on the fuel economy, emissions, ageing of components and vehicle drive-ability. It is generally accepted that Stochastic Dynamic Programming (SDP) can be used to produce a near-optimal control strategy provided that an accurate Markov model of the drive-cycle is available, and the cost function used for the optimisation is representative of the true running cost of the vehicle. The vast majority of research in this field focussing solely on the optimisation of the fuel economy, however for a fuel cell hybrid vehicle, the degradation of the fuel cell contributes significantly to the overall running cost of the vehicle, and should therefore be included in calculation of the running cost during the optimisation process. In this work, an optimised controller using SDP is developed for a campus passenger vehicle in order to minimise the lifetime cost of both fuel consumption and fuel cell degradation. The vehicle is then simulated over a number of typical journey’s obtained from data logging during its use on the University of Birmingham’s campus. It is shown that the expected lifetime cost due to fuel cell degradation massively outweighs the cost of the fuel consumed.

A comparison study of different semi-active hybrid energy storage system topologies for electric vehicles

In this paper, four different semi-active hybrid energy storage systems (HESSs), which use both supercapacitors (SCs) and batteries, are compared based on an electric city bus running the China Bus Driving Cycle (CBDC). The SC sizes of the different HESS topologies are optimized by using the dynamic programming (DP) approach, based on a dynamic degradation model of the LiFePO4 battery. The operation costs of different HESSs, including the electricity and the battery degradation costs over a whole CBDC, are minimized in the optimization process. Based on the DP results, near-optimal control strategies of different HESSs for on-line uses are proposed. Finally, the four HESS topologies are comprehensively compared from different aspects, including operation cost, initial cost, and DC bus voltage variation. Simulation results show that all HESS topologies have their merits and drawbacks, and can be used in different applications with different requirements. In addition, about 50% of the operation cost of the energy storage system is reduced by the semi-active HESSs when compared to the battery-only topology. Thus the effectiveness of adopting the SC in the HESS is verified.