Online Optimal Control Research Articles

Online optimal consensus control of unknown linear multi-agent systems via time-based adaptive dynamic programming

This paper considers the online optimal consensus control problem for unknown linear discrete-time (DT) multi-agent systems (MASs). Based on time-based adaptive dynamic programming (ADP) method, the control policies are designed by utilizing the current and recorded data of unknown MASs. The critic-actor NN frameworks are employed to approximate the performance indexes and optimal control policies, respectively. The NN weights are updated once at the sampling instant to produce real-time online control. Furthermore, the control policies are proved to effectively drive the MASs to achieve consistency and satisfy the Nash equilibrium. Finally, a numerical example is implemented to shown the feasibility of the control scheme.

Phase portraits as movement primitives for fast humanoid robot control

Currently, usual approaches for fast robot control are largely reliant on solving online optimal control problems. Such methods are known to be computationally intensive and sensitive to model accuracy. On the other hand, animals plan complex motor actions not only fast but seemingly with little effort even on unseen tasks. This natural sense to infer temporal dynamics and coordination motivates us to approach robot control from a motor skill learning perspective to design fast and computationally light controllers that can be learned autonomously by the robot under mild modeling assumptions. This article introduces Phase Portrait Movement Primitives (PPMP), a primitive that predicts dynamics on a low dimensional phase space which in turn is used to govern the high dimensional kinematics of the task. The stark difference with other primitive formulations is a built-in mechanism for phase prediction in the form of coupled oscillators that replaces model-based state estimators such as Kalman filters. The policy is trained by optimizing the parameters of the oscillators whose output is connected to a kinematic distribution in the form of a phase portrait. The drastic reduction in dimensionality allows us to efficiently train and execute PPMPs on a real human-sized, dual-arm humanoid upper body on a task involving 20 degrees-of-freedom. We demonstrate PPMPs in interactions requiring fast reactions times while generating anticipative pose adaptation in both discrete and cyclic tasks.

Analytical approach to online optimal control strategy of energy storage devices in energy system

• Proposing analytical method for global optimal strategy of battery in energy system. • Reducing calculation burden of battery scheduling in energy systems. • Sensitivity analysis of energy system performance in optimal operation mode. This paper presents an analytical approach for the optimal operation of battery-integrated energy systems (BIES). Energy profile sectionalizing, achieved through the offline studies, is proposed in this paper as a framework for optimal energy dispatch in BIES, and an online strategy is then explained to control the energy system optimally. In this paper, a general framework of the battery control is addressed with considering battery characteristics and system electricity tariffs to make the proposed method applicable to real cases. Applying the proposed method to BIES shows great advantages in optimal battery scheduling as it guarantees to achieve the global optimal operation of the battery considering its actual parameters compared to the previous studies focusing on the use of optimization methods whose outputs are near-optimal solutions. Real load profile and electricity prices are used to test the proposed approach through a developed code in MATLAB software. The results show that the cost of the energy system and calculation burden of achieving optimal results are decreased compared with using the genetic algorithm (GA) and harmony search algorithm (HSA).

Optimal polymer grade transitions for fluidized bed reactors

Industrial polymerization plants experience frequent changes of products, driven by end-use properties to meet various market requirements. Efficient grade transition policies are essential to save time and materials. In this study, the gas-phase catalytic polymerization is modeled in a fluidized bed reactor by a single-phase model, and dynamic optimization is implemented to determine optimal operating sequences for grade changes. Two optimization formulations, a single-stage and a multi-stage formulation, are introduced and compared. The superiority of the multi-stage formulation is demonstrated owing to a better control on each stage during the transition and a further reduction of off-grade time. Subsequently, an on-line optimal control framework is established by incorporating shrinking horizon nonlinear model predictive control with an expanding horizon weighted least-square estimator for process states and unknown parameters. The results of a case study indicate the designed framework is able to handle process uncertainty, while reducing the transition time.

Model-based multi-objective predictive scheduling and real-time optimal control of energy systems in zero/low energy buildings using a game theory approach

Online optimal control of energy systems plays a significant role in achieving the zero/low energy goal and high energy efficiency for zero/low energy buildings. Increasing amount of studies are conducted on the multi-objective optimal control of building energy systems in recent years since multiple objectives are often concerned in practice. The weighted sum method is widely used to solve the multi-objective optimization problems for online optimal controls. However, it is often difficult and even impractical to determine proper weights for objectives of different natures. Existing studies on online optimal control of energy systems in zero/low energy buildings focused on the predictive scheduling, and very few studies addressed the real-time optimal control aspect which is also essential in operation. In this study, a coordinated online multi-objective control strategy, consisting of two control optimization schemes, is therefore proposed for predictive scheduling and real-time optimal control of energy systems in zero/low energy buildings. A cooperative game theory-based method is adopted for the online multi-objective optimizations. The control strategy was tested and evaluated via simulation of the energy systems in a zero energy building with battery storage on two typical days. Control variables and weights of objectives were optimized to minimize the combined optimization objective involving energy cost and grid impact. The test results show that it is essential and beneficial to coordinate the predictive scheduling and real-time optimal control in actual operation. The cooperative game theory-based method is effective for the online multi-objective optimization without the need of setting weights of different objectives.

Model-Free Fast Frequency Control Support With Energy Storage System

With the advantage of quick response and flexible ramp, energy storage system (ESS) offers a promising capability of fast frequency control for power systems, especially under a severe disturbance. This paper proposes an ESS control strategy using local measurement in order to provide fast frequency support right after a sufficiently severe disturbance is observed. The proposed strategy includes online frequency characteristic estimator and online optimization controller. Distinguished from existing approaches, no prior knowledge of power system model is required in this work, including generator configuration, operation condition, and frequency response of each component. Instead, the overall system frequency characteristic is approximated online by the proposed estimator. Then, the power setpoint of ESS is determined adaptively to participate in frequency control by the proposed model predictive controller. Finally, the effectiveness of the proposed method is verified via real-world data of a provincial power grid in China.

Risk-based online robust optimal control of air-conditioning systems for buildings requiring strict humidity control considering measurement uncertainties

The total floor area and energy consumption of buildings or spaces requiring strict temperature and humidity control have been growing rapidly worldwide. A major challenge for achieving energy-efficient control of air-conditioning systems in such applications is the measurement uncertainties underlying the systems’ online optimal control decisions under ever-changing working conditions. This paper proposes a risk-based online robust optimal control strategy for multi-zone air-conditioning systems considering component performance degradation and measurement uncertainties. A risk-based online control decision scheme, as the core of the strategy, is developed for decision-making by compromising the failure risks and energy benefits of different control modes considering uncertainties in the information used. The proposed strategy is tested and implemented in a simulation platform based on an existing pharmaceutical industrial building. The results show that the proposed strategy made the optimal online control decisions, allowing for the measurement uncertainties. Compared with a commonly used control strategy, the proposed strategy achieved approximately 20% overall energy saving in the test period.

Online Optimal Mode Control for Plug-in Hybrid Vehicles Based on Driving Routes

This paper proposes an online optimal mode control method to minimize fuel consumption for plug-in hybrid vehicles (PHVs) considering two drive modes: electric vehicle (EV) and hybrid vehicle (HV) modes. The proposed method predicts fuel and electricity consumption of PHVs based mainly on a driving-route model that considers road grades and vehicle speed distributions. The driving-route model is estimated with terrain maps and historical driving data on a planned route. In this work, two energy consumption maps are built for the EV and HV modes of the PHVs. The driving-route model and energy consumption maps lead to the formulation of an integer linear programming problem by regarding the two drive modes as binary variables. A detailed vehicle simulator, called ADVISOR, demonstrates that the proposed method improves fuel efficiency over that of a conventional method.

Real-time online optimal control of current-fed dual active bridges based on machine learning

This paper proposes a real-time online optimal (RT-OPT) control method based on machine learning for a current-fed dual active bridge (CF-DAB). The basis of this control strategy is the linear quadratic optimal control, which designs the sliding surface and realizes power control based on sliding mode control (SMC). For the parameters of Q and R in the objective function of the linear quadratic regulator (LQR), a genetic algorithm is used to find the optimal value, and the optimal value is taken as the sample data. Through machine learning offline training, a neural network is obtained and run online to realize real-time online optimal control. The control method was verified by simulations in MATLAB/Simulink. The RT-OPT method achieves the expected functionality, and has better dynamic and steady-state performance than the PI controller.

Adaptive Energy Management Strategy for Extended-Range Electric Vehicle Based on Micro-Trip Identification

In order to improve the fuel economy of an extended-range electric vehicle with the engine-generator, an adaptive energy management strategy has been proposed in this paper. First, micro-trip decomposition analysis of the standard driving cycles are conducted, and these micro trips are classified as four kinds of driving patterns by K-means clustering method. Second, an optimal energy allocation for the engine-generator and battery is designed by Pontryagin’s minimum principle (PMP). The proposed approach should realize the energy management and maintain the battery in a charge sustaining mode. Third, an online optimal control is conducted by a micro-trip identification algorithm. By utilizing the clustering driving patterns, the adaptive energy management strategy is achieved by the selection of optimal PMP co-state variable. Finally, the experimental performance comparisons for different control strategies and driving cycles are investigated to shown the efficiency of proposed controller.

On-Line PID Parameters Optimization Control for Wind Power Generation System Based on Genetic Algorithm

An on-line PID parameter optimization control for the wind power generation system based on a genetic algorithm is proposed in this paper. Firstly, the anti-saturation PID control strategy is involved with considering the instability and complexity of the wind power source. Further, a genetic algorithm is introduced for an on-line optimization of the PID parameters. The simulation studies are carried out on a control model of wind power, using MATLAB simulation system. It is demonstrated that the proposed control strategy can not only solve the integral saturation, but also suppress the harmonics in the output waveform and enhance the power factor of system.

Optimization Method for the Energy and Emissions Management of a Hybrid Electric Vehicle with an Exhaust Aftertreatment System

This paper presents a real-time optimization method to compute the fuel-optimal torque split, gear selection and engine on/off command for a Diesel hybrid electric vehicle equipped with an exhaust aftertreatment system. We aim to minimize the amount of fuel consumed, while achieving a charge-sustaining operation and keeping the tailpipe NOx emissions below the legislative limit. We simplify the full vehicle model to facilitate the formulation of a mixed-integer convex problem which is then solved using the proposed iterative convex optimization (ICO) algorithm. We validate the result by comparing it to the globally optimal solution computed using dynamic programming (DP). For the simple model, the ICO algorithm finds the same solution as the DP benchmark. The computation time was reduced from one week for the DP benchmark to 49s for the ICO solution. By comparing the DP solution obtained on the full model with the ICO solution evaluated on the full model, we observe an offset in the solution due to model mismatch, but find that the ICO algorithm captures the qualitative trends of the optimal solution. The proposed algorithm is capable of solving the energy and emissions management problem in real-time, forming the basis for online optimal control.

Tuning heuristics and convergence analysis of reinforcement learning algorithm for online data-based optimal control design

A heuristic for tuning and convergence analysis of the reinforcement learning algorithm for control with output feedback with only input / output data generated by a model is presented. To promote convergence analysis, it is necessary to perform the parameter adjustment in the algorithms used for data generation, and iteratively solve the control problem. A heuristic is proposed to adjust the data generator parameters creating surfaces to assist in the convergence and robustness analysis process of the optimal online control methodology. The algorithm tested is the discrete linear quadratic regulator (DLQR) with output feedback, based on reinforcement learning algorithms through temporal difference learning in the policy iteration scheme to determine the optimal policy using input / output data only. In the policy iteration algorithm, recursive least squares (RLS) is used to estimate online parameters associated with output feedback DLQR. After applying the proposed tuning heuristics, the influence of the parameters could be clearly seen, and the convergence analysis facilitated.

Hardware-in-the-Loop Iterative Optimal Feedback Control Without Model-Based Future Prediction

Optimal control provides a systematic approach to control robots. However, computing optimal controllers for hardware-in-the-loop control is sensitively affected by modeling assumptions, computationally expensive in online implementation, and time-consuming in practical application. This makes the theoretical appeal of optimization challenging to exploit in real-world implementation. In this paper, we present a novel online optimal control formulation that aims to address the above-mentioned limitations. The formulation combines a model with measured state information to efficiently find near-optimal feedback controllers. The idea to combine a model with measurements from the actual motion is similar to what is used in model predictive control formulations, with the difference that here the model is not used for future prediction, the optimization is performed along the measured trajectory of the system, and the online computation is reduced to a minimum; it requires a small-scale, one time step, static optimization, instead of a large-scale, finite time horizon, dynamic optimization. The formulation can be used to solve optimal control problems defined with nonlinear cost, nonlinear dynamics, and box-constrained control inputs. Numerical simulations and hardware-in-the-loop experiments demonstrate the effectiveness of the proposed hardware-in-the-loop optimal control approach.

Assessment and Control of Microgrid Impacts on Distribution Networks by Using Experimental Tests

The main aim of this article is to assess and control impacts that a grid-tied microgrid may have on a distribution network. For this purpose, the tie-line power flow has been considered in a performance index to be minimized. In particular, three statistical indices have been compared on the microgrid operated under a classical day-ahead dispatching policy and an online optimal control strategy. The online optimization process starts from the same day-ahead dispatching policy and tries to compensate the tie-line power flow deviations from the scheduled value. For this purpose, a real-time economic dispatching algorithm based on a proportional integral derivative controller has been developed and implemented on the PrInCE Lab microgrid, located at the Polytechnic University of Bari, Bari, Italy.

Full-Body Optimal Control Toward Versatile and Agile Behaviors in a Humanoid Robot

In this letter, we develop an optimal control framework that takes the full-body dynamics of a humanoid robot into account. Employing full-body dynamics has been explored in, especially, an online optimal control approach known as model predictive control (MPC). However, whole-body motions cannot be updated in a short period of time due to MPC's large computational burden. Thus, MPC has generally been evaluated with a physical humanoid robot in a limited range of tasks where high-speed motion executions are unnecessary. To cope with this problem, our multi-timescale control framework drives whole-body motions with a computationally efficient hierarchical MPC. Meanwhile, a biologically inspired controller maintains the robot's posture for a very short control period. We evaluated our framework in skating tasks with simulated and real lower-body humanoids that have rollers on the feet. Our simulated robot generated various agile motions such as jumping over a bump and flipping down from a cliff in real time. Our real lower-body humanoid also successfully generated a movement down a slope.

Online optimal control for dynamic positioning of vessels via time-based adaptive dynamic programming

In this paper, a discrete-time online optimal control via time-based adaptive dynamic programming is developed for dynamic positioning (DP) of vessels in the presence of unknown system dynamic, energy conservation and wastage decrease of equipment. Firstly, a model network is established by a neural network (NN) to identify the DP system. And then, a NN optimal control scheme is developed, in which current and recorded data of DP vessel are utilized to train the critic network and action network. The optimal cost and control law are updated once at the sampling instant. The designed DP optimal control policy can maintain the vessel at desired position and heading angle, and guarantee the uniform ultimate boundedness of all the signals in the closed-loop system simultaneously. Finally, simulation results involving a supply vessel demonstrate the effectiveness of the proposed DP optimal control law.

Machine learning-based modeling and operation for ALD of SiO2 thin-films using data from a multiscale CFD simulation

Atomic layer deposition (ALD) is a widely utilized deposition technology in the semiconductor industry due to its superior ability to generate highly conformal films and to deposit materials into high aspect-ratio geometric structures. However, ALD experiments remain expensive and time-consuming, and the existing first-principles based models have not yet been able to provide solutions to key process outputs that are computationally efficient, which is necessary for on-line optimization and real-time control. In this work, a multiscale data-driven model is proposed and developed to capture the macroscopic process domain dynamics with a linear parameter varying model, and to characterize the microscopic domain film growth dynamics with a feed-forward artificial neural network (ANN) model. The multiscale data-driven model predicts the transient deposition rate from the following four key process operating parameters that can be manipulated, measured or estimated by process engineers: precursor feed flow rate, operating pressure, surface heating, and transient film coverage. Our results demonstrate that the multiscale data-driven model can efficiently characterize the transient input-output relationship for the SiO2 thermal ALD process using bis(tertiary-butylamino)silane (BTBAS) as the Si precursor. The multiscale data-driven model successfully reduces the computational time from 0.6 to 1.2h for each time step, which is required for the first-principles based multiscale computational fluid dynamics (CFD) model, to less than 0.1s, making its real-time usage feasible. The developed data-driven modeling methodology can be further generalized and used for other thermal ALD or similar deposition systems, which will greatly enhance the feasibility of industrial manufacturing processes.

Real-Time Switched Model Predictive Control for a Cyber-Physical Wind Turbine Emulator

The high complexity and nonlinearity of wind turbine (WT) systems impose the utilization of rigorous control methods such as model predictive control (MPC). MPC algorithms are computationally intensive requiring investigation of real-time implementability and feasibility in addition to control performance metrics. In this article, a switched model predictive controller (SMPC) is developed, implemented, and investigated for control objectives performance and real-time metrics. This article has two main contributions. First, embedded real-time SMPC is developed using qpOASES as an embedded solver for the online optimal control problem. Second, a cyber-physical real-time emulator for variable-speed variable-pitch utility-scale WT is developed and implemented on an xPC target machine using a high-fidelity linear parameter-varying model. The SMPC is evaluated on the fatigue, aerodynamics, structures, and turbulence (FAST) design code and the real-time emulator. The analysis and investigation of results highlight the feasibility and capability of SMPC for handling control objectives of WT systems within real-time using short control periods.

Reinforcement-Learning-Based Optimal Control of Hybrid Energy Storage Systems in Hybrid AC–DC Microgrids

In this paper, a reinforcement-learning-based online optimal (RL-OPT) control method is proposed for the hybrid energy storage system (HESS) in ac-dc microgrids involving photovoltaic systems and diesel generators (DGs). Due to the low system inertia, conventional unregulated charging and discharging (C&D) of energy storages in microgrids may introduce disturbances that degrade the power quality and the system performance, especially in fast C&D situations. Secondary and tertiary control levels can optimize the state of charge reference of HESS; however, they are lacking the direct controllability of regulating the transient performance. Additionally, the uncertainties in practical systems greatly limit the performance of conventional model based controllers. In this study, the optimal control theory is applied to optimize the C&D profile and to suppress the disturbances caused by integrating HESS. Neural networks are devised to estimate the nonlinear dynamics of HESS based on the input/output measurements, and to learn the optimal control input for bidirectional-converter-interfaced HESS using the estimated system dynamics. Because the proposed RL-OPT method is fully decentralized, which only requires the local measurements, the plug and play capability of HESS can be easily realized. Both islanded and grid-tied modes are considered. Extensive simulations and experiments are conducted to evaluate the effectiveness of the proposed method.