Near-optimal Control Research Articles

Online Nash Solution in Networked Multirobot Formation Using Stochastic Near-Optimal Control Under Dynamic Events.

This article proposes an online stochastic dynamic event-based near-optimal controller for formation in the networked multirobot system. The system is prone to network uncertainties, such as packet loss and transmission delay, that introduce stochasticity in the system. The multirobot formation problem poses a nonzero-sum game scenario. The near-optimal control inputs/policies based on proposed event-based methodology attain a Nash equilibrium achieving the desired formation in the system. These policies are generated online only at events using actor-critic neural network architecture whose weights are updated too at the same instants. The approach ensures system stability by deriving the ultimate boundedness of estimation errors of actor-critic weights and the event-based closed-loop formation error. The efficacy of the proposed approach has been validated in real-time using three Pioneer P3-Dx mobile robots in a multirobot framework. The control update instants are minimized to as low as 20% and 18% for the two follower robots.

Learning and Near-Optimal Control of Underactuated Surface Vessels With Periodic Disturbances.

In this article, we propose a novel learning and near-optimal control approach for underactuated surface (USV) vessels with unknown mismatched periodic external disturbances and unknown hydrodynamic parameters. Given a prior knowledge of the periods of the disturbances, an analytical near-optimal control law is derived through the approximation of the integral-type quadratic performance index with respect to the tracking error, where the equivalent unknown parameters are generated online by an auxiliary system that can learn the dynamics of the controlled system. It is proved that the state differences between the auxiliary system and the corresponding controlled USV vessel are globally asymptotically convergent to zero. Besides, the approach theoretically guarantees asymptotic optimality of the performance index. The efficacy of the method is demonstrated via simulations based on the real parameters of an USV vessel.

PBN-Based Time-Optimal Terminal Air Traffic Control Using Cellular Automata

Over the last decade, there has been an increase in air transportation, which has raised awareness to provide necessary procedures to respond to air transportation demands and reduce delay costs. One such procedure, “performance-based navigation (PBN),” currently studied, is expected to increase the airspace capacity based on aircraft's individual performances. This research describes an efficient and expandable model for allocating standard terminal arrival route during continuous descent approach (CDA) that effectively uses PBN requirements. As a case study, we have considered the “Atlanta International Airport” and verified the model's accuracy using real-time traffic data. The model is then used to study mixed traffic behavior concerning each aircraft's navigation capability. We show that our technique helps decrease time separation minima, reducing traffic delays by up to 50% during busy periods. Furthermore, we show that our approach could enhance target traffic flow by as much as 25% while maintaining proper separation minima. Finally, we show that near-optimal control advisory could also be obtained to minimize the target traffic flow's total delay.

Near-optimal control and threshold behavior of an avian influenza model with spatial diffusion on complex networks.

Near-optimization is as sensible and important as optimization for both theory and applications. This paper concerns the near-optimal control of an avian influenza model with saturation on heterogeneous complex networks. Firstly, the basic reproduction number $ \mathcal{R}_{0} $ is defined for the model, which can be used to govern the threshold dynamics of influenza disease. Secondly, the near-optimal control problem was formulated by slaughtering poultry and treating infected humans while keeping the loss and cost to a minimum. Thanks to the maximum condition of the Hamiltonian function and the Ekeland's variational principle, we establish both necessary and sufficient conditions for the near-optimality by several delicate estimates for the state and adjoint processes. Finally, a number of examples presented to illustrate our theoretical results.

Optimizing the Operation of a Hybrid Ground Source Heat Pump System Under Uncertainty

Hybrid renewable energy systems that combine conventional heating, ventilation, and air conditioning (HVAC) systems and ground source heat pumps (GSHP) have become an attractive alternative for conventional HVAC systems due to their higher cost and energy efficiency.Furthermore, control strategies that exploit predictive information about weather and building occupants' activity can further reduce the system operating costs. This study proposes a Stochastic Model Predictive Control (SMPC) for hybrid GSHP systems considering the stochastic nature of the building space heating demand. In SMPC, near-optimal control decisions are found for the current and future states of the system through the application of Regression Monte-Carlo techniques. We compare the performance of SMPC to that gained by using setpoint Control (SPC) and Model Predictive Control (MPC) which uses a deterministic forecast. It is found that by taking uncertainty into account via SMPC, the operating cost reduction compared to SPC is approximately equal to half of the cost-optimality gap between SPC and an idealized controller that is represented by MPC with perfect future information. Furthermore, we find that MPC using a forecast based on expected values leads to greater operating costs compared to the simpler SPC strategy when variability and uncertainty are present.

Joint Traffic-Aware Consolidated Middleboxes Selection and Routing in Distributed SDNs

Software middlebox-based services can be flexibly managed by software defined networking (SDN) and network function virtualization (NFV). Meanwhile, traffic routing can be simplified and the number of routing rules in the SDN-enabled switches can be reduced through the consolidated middlebox model. However, different network functions in middleboxes may alter the volume of processed traffic, so high congestion may occur in specific bottleneck links if middlebox selection and traffic routing are not well jointly planned. Besides, in a statically switch-controller configured SDN, traffic dynamics will not only affect the link load in the data plane, but also pose a challenge to controller load balancing. Therefore, it’s necessary to achieve better quality-of-service (QoS) performance in both control and data planes. In this article, we first formulate this problem as a joint traffic-aware consolidated middleboxes selection and routing (JTMSR) problem and prove its NP-hardness. Then, we design a two-phase algorithm to achieve the controller and link load balancing where the first phase is to redirect selected flows by applying wildcard rules and the second phase is to find fine-grained routing path by a rounding-based algorithm with bounded approximation factor. Finally, compared with the existing algorithms through extensive simulations, it demonstrates that our method has near-optimal controller load balancing and link load balancing performance and can improve response time by 9.7% compared with static scheme.

A near-optimal decentralized control approach for polynomial nonlinear interconnected systems

This article tackles the decentralized near-optimal control problem for the class of nonlinear polynomial interconnected system based on a shifted Legendre polynomials direct approach. The proposed method converts the interconnected optimal control problems into a nonlinear programming one with multiple constraints. In light of the formulated NLP optimization, state and control coefficients are used to design a nonlinear decentralized state feedback controller. Overall closed-loop system stability sufficient conditions are investigated with the help of Grönwall lemma. The triple inverted pendulum case is considered for simulation. Satisfactory results are obtained in both open-loop and closed-loop schemes with comparison to collocation and state-dependent Riccati equation techniques.

An Online Event-Triggered Near-Optimal Controller for Nash Solution in Interconnected System.

This article proposes a real-time event-triggered near-optimal controller for the nonlinear discrete-time interconnected system. The interconnected system has a number of subsystems/agents, which pose a nonzero-sum game scenario. The control inputs/policies based on proposed event-based controller methodology attain a Nash equilibrium fulfilling the desired goal of the system. The near-optimal control policies are generated online only at events using actor-critic neural network architecture whose weights are updated too at the same instants. The approach ensures stability as the event-triggering condition for agents is derived using Lyapunov stability analysis. The lower bound on interevent time, boundedness of closed-loop parameters, and optimality of the proposed controller are also guaranteed. The efficacy of the proposed approach has been validated on a practical heating, ventilation, and air-conditioning system for achieving the desired temperature set in four zones of a building. The control update instants are minimized to as low as 27% for the desired temperature set.

Near-optimal control for a stochastic multi-strain epidemic model with age structure and Markovian switching

ABSTRACT Stochastic noise can affect the dynamics of infectious diseases. In this paper, a new stochastic multi-strain epidemic model with white noise, Markovian switching, vaccination control and age structure is established. Some priori estimates of the susceptible, infected and recovered are obtained. Under the condition of the objective function is minimised, we formulate sufficient and necessary conditions for the near-optimal control problems of the system, which is using a nearly maximum condition of the Hamiltonian function. The great influence of age structure, white noise, Markovian switching and vaccination control on the dynamical behaviour of infectious diseases and the existence of near-optimal control are demonstrated by numerical simulations.

Dynamic pricing and fleet management for electric autonomous mobility on demand systems

The proliferation of ride sharing systems is a major drive in the advancement of autonomous and electric vehicle technologies. This paper considers the joint routing, battery charging, and pricing problem faced by a profit-maximizing transportation service provider that operates a fleet of autonomous electric vehicles. We first establish the static planning problem by considering time-invariant system parameters and determine the optimal static policy. While the static policy provides stability of customer queues waiting for rides even if consider the system dynamics, we see that it is inefficient to utilize a static policy as it can lead to long wait times for customers and low profits. To accommodate for the stochastic nature of trip demands, renewable energy availability, and electricity prices and to further optimally manage the autonomous fleet given the need to generate integer allocations, a real-time policy is required. The optimal real-time policy that executes actions based on full state information of the system is the solution of a complex dynamic program. However, we argue that it is intractable to exactly solve for the optimal policy using exact dynamic programming methods and therefore apply deep reinforcement learning to develop a near-optimal control policy. The two case studies we conducted in Manhattan and San Francisco demonstrate the efficacy of our real-time policy in terms of network stability and profits, while keeping the queue lengths up to 200 times less than the static policy.

Onboard Generation of Optimal Trajectories for Hypersonic Vehicles Using Deep Learning

Recent development of deep learning has shown that a deep neural network (DNN) is capable of learning the underlying nonlinear relationship between the state and the optimal actions for nonlinear optimal control problems. In terms of hypersonic flight, this suggests that the DNN-based trajectory controller may be considered to take over all or part of the onboard trajectory generation and guidance system. This Paper investigates the possibility of training the DNN-based controller off line using the optimal state-action samples obtained from high-fidelity algorithms. The time-consuming computation is carried out off line, and the resulting DNN-based controller is potentially capable of near-optimal control with real-time performance and stable convergence. First, the system dynamics of hypersonic flight are described, and the trajectory optimization problem is formulated as a highly nonlinear optimal control problem. Then, the state-action vectors are extracted from the optimal trajectories generated by solving the formulated optimal control problem from random initial states using a homotopy method. Thereafter, DNNs are designed to learn the functional relationship between the flight states and the optimal actions to enable the capability of optimal action predictions. At last, the performance of the proposed approach is demonstrated through numerical simulations.

Synchronous Fault-Tolerant Near-Optimal Control for Discrete-Time Nonlinear PE Game.

In this article, the synchronous fault-tolerant near-optimal control strategy design problem is studied for a class of discrete-time nonlinear pursuit-evasion (PE) games. In the studied PE game, the input saturation phenomenon and possible actuator fault are simultaneously taken into consideration. To accelerate the estimation speed, a novel nonlinear fault estimator is designed by introducing a nonlinear function. Then, for the purpose of obtaining the synchronous control strategy for the discrete-time PE games, an approximate Hamilton-Jacobi-Isaacs (HJI) equation is established, which is seldom utilized for the discrete-time approximate dynamic programming in most existing results. It should be noticed that the synchronous control strategy designed based on the approximate HJI equation can be convergent very fast because of its quasi-Newton's iteration form. Furthermore, the sufficient condition is established to guarantee that the studied system is uniformly ultimately bounded. Finally, a numerical simulation of the hypersonic vehicle system is carried out to validate the proposed methodology.

Event-Triggered Adaptive Critic Control Design for Discrete-Time Constrained Nonlinear Systems

In this paper, through event-triggered approach, the constrained near-optimal control problem for a class of nonlinear discrete-time systems is investigated and solved by heuristic dynamic programming (HDP) technique. The proposed method can reduce the amount of computation remarkably without deteriorating the system stability. In order to overcome the control constraints and reduce the computational burden, a nonquadratic performance index is introduced. Then, stability analysis of the event-triggered system with control constraints and an event-triggered constrained controller design algorithm are given. Three neural networks are used in the HDP scheme, which are designed to identify the unknown nonlinear system, approximate value function, and control law, respectively. In the model neural network, an effective method is developed to initialize its weights. Finally, two examples are included to demonstrate the present method.

Artificial intelligence control of a turbulent jet

An artificial intelligence (AI) control system is developed to maximize the mixing rate of a turbulent jet. This system comprises six independently operated unsteady minijet actuators, two hot-wire sensors placed in the jet, and genetic programming for the unsupervised learning of a near-optimal control law. The ansatz of this law includes multi-frequency open-loop forcing, sensor-feedback and nonlinear combinations thereof. Mixing performance is quantified by the decay rate of the centreline mean velocity of jet. Intriguingly, the learning process of AI control discovers the classical forcings, i.e. axisymmetric, helical and flapping achievable from conventional control techniques, one by one in the order of increased performance, and finally converges to a hitherto unexplored forcing. Careful examination of the control landscape unveils typical control laws, generated in the learning process, and their evolutions. The best AI forcing produces a complex turbulent flow structure that is characterized by periodically generated mushroom structures, helical motion and oscillating jet column, all enhancing the mixing rate and vastly outperforming others. Being never reported before, this flow structure is examined in various aspects, including the velocity spectra, mean and fluctuating velocity fields and their downstream evolution, and flow visualization images in three orthogonal planes, all compared with other classical flow structures. Along with the knowledge of the minijet-produced flow and its effect on the initial condition of the main jet, these aspects cast valuable insight into the physics behind the highly effective mixing of this newly found flow structure. The results point to the great potential of AI in conquering the vast opportunity space of control laws for many actuators and sensors and in optimizing turbulence.

A near-optimal control algorithm for central cooling plants with electric and/or gas-driven chillers

This paper presents a near-optimal control algorithm for minimizing energy costs in hybrid cooling plants in response to cooling loads. The approach is based upon an approximate solution to the partial differential equations involved in minimizing the cost of energy consumed by the plant with respect to two control variables: cooling tower airflow and condenser water flow. The parameters of the algorithm can be determined with design information of the chillers and cooling towers, along with some measurements of total condenser water flow and pump power consumption. In addition to reducing plant operating costs, the algorithm simplifies tower control and is more stable compared with conventional tower control strategies such as constant condenser water supply temperature or constant approach to wet-bulb. A large-scale cooling plant model was utilized to evaluate the performance of the approach compared with optimal control and two heuristic control strategies over six months. The difference between the energy costs associated with the near-optimal control approach and optimal control was 1.7%. Comparison with the heuristic control strategies shows that significant energy savings can be achieved with the proposed algorithm.

Maximum Principle for Near-Optimality of Mean-Field FBSDEs

The present paper concerns with a near-optimal control problem for systems governed by mean-field forward-backward stochastic differential equations (FBSDEs) with mixed initial-terminal conditions. Utilizing Ekeland’s variational principle as well as the reduction method, the necessary and sufficient near-optimality conditions are established in the form of Pontryagin’s type. The results are obtained under restriction on the convexity of the control domain. As an application, a linear-quadratic stochastic control problem is solved explicitly.

Optimal insertion of customers with waiting time targets

The insertion of randomly-arriving high-priority customers (HCs) into existing queues leads to longer waiting time of regular or scheduled customers. However, given their different levels of priority, not all HCs need to be served immediately. To reduce the waiting time for regular customers without affecting the service quality for HCs, this paper addresses the optimal insertion problem of HCs by considering their waiting time targets assigned based on their levels of priority. A finite-horizon Markov decision process model is proposed to minimize the total waiting cost incurred by both regular customers and HCs. The marginal waiting penalty is constant for regular customers and non-decreasing for HCs. Several properties are observed: the optimal control policy is proved to be a state-dependent threshold policy; the marginal effect of each state variable on the optimal action is bounded by 1; and, in some meaningful cases, the threshold proves to be state-independent. Based on these properties, several heuristic policies are proposed to solve large-size problems. Numerical experiments are performed to validate the structure of the optimal control policies and compare the performances of the heuristic policies. These results imply that the optimal control policy significantly outperforms the traditional HC-first policy. The best heuristic policy, characterized by a threshold policy derived through policy iteration, performs within 0.44% of an optimal policy for most cases, which offers insights into the near-optimal control for large-size problems.

Neural network optimal control in astrodynamics: Application to the missed thrust problem

While high-efficiency propulsion techniques are enabling new mission concepts in deep space exploration, their limited thrust capabilities necessitate long thrusting arcs and make spacecraft more susceptible to missed thrust events. To correct for such mishaps, most spacecraft require updated trajectories that are relayed from Earth. While this solution is viable for spacecraft near Earth, in deep space, where one-way communication time is measured in hours, a delay in transmission may prolong the time of flight or result in a complete loss of mission. Such problems can be alleviated by increasing the spacecraft's onboard autonomy in guidance. This paper demonstrates how a computationally lightweight neural network can map the spacecraft's state to a near-optimal control action, autonomously guiding a spacecraft within different astrodynamic regimes and optimality criteria. The neural network is trained using supervised learning and datasets comprised of optimal state-action pairs, as determined through traditional direct and indirect methods. Additionally, the neural network-designed solutions retain optimality and time of flight corresponding to traditional trajectories. Finally, the same neural networks can autonomously correct for most missed thrust events encountered on long-duration low-thrust trajectories. The presented results provide a path for mitigating risks associated with the use of high-efficiency low-thrust propulsion techniques.

Transfer Learning Applied to Reinforcement Learning-Based HVAC Control

Modern control solutions for HVAC have demonstrated excellent cost and energy savings through the utilisation of machine learning techniques. However, a challenging problem faced by most machine learning tasks is the amount of time and data required to train effective policies in the absence of prior knowledge. Considering that buildings from a specific geographical location share common environmental and structural features, this paper investigates the impact of spatial changes on performance accuracy through the use of transfer learning applied to reinforcement learning based HVAC control. We propose the development of an adapted RL (Q-learning) algorithm which can transfer HVAC control polices, adjusting themselves according to spatial changes. We examine the performance of our approach across multiple different locations. Moreover, an analysis of the user’s time out comfort has been made, comparing models with and without transfer learning. The results from different cases show that after applying transfer learning the learning time to train optimal or near-optimal control policies was reduced by more than a factor of 6 when comparing to experiments without it. Furthermore, the test case where the spatial variation was lower than 50% achieved a similar performance for both dynamic and static HVAC control, presenting an average time out comfort error of 2.55% and 3.83%, respectively. From the user’s perspective, it means they will not feel any additional discomfort, as the number of minutes out of the comfort zone for the static version is approximately the same for a 1-day interval. Finally, when examining the effect of transfer learning on geographical changes, the proposed method demonstrated higher performance in countries where the temperature variation is lower, reducing time out comfort by one-third. If an agent receives a policy from a place where the environmental conditions are very different the proposed method will still work and find the best policy, but not as fast as receiving it from a similar place.

Iterative Online Optimal Feedback Control

This article proposes a data-driven iterative feedback control method to efficiently solve finite time horizon, nonlinear, input constrained optimal control problems. The proposed method introduces a novel approach to combine an inexact system model with measured state information to reduce the cost and provide near-optimal control by approximately solving the optimal control problem along the trajectory of the real system, as opposed to solving it along the trajectory predicted by the inexact model. We present a new algorithm that implements the proposed method, establish the convergence and optimality properties of the proposed algorithm, and compare it to optimal feedback control and model predictive control that solve the same optimal control problem along the trajectory predicted by the inexact model. Finally, we illustrate the generality of the proposed algorithm by approximately solving a challenging optimal control problem with unknown and changing dynamics.