Model Predictive Control Policy Research Articles

Hybrid Control Policy for Artificial Pancreas Via Ensemble Deep Reinforcement Learning.

The artificial pancreas (AP) has shown promising potential in achieving closed-loop glucose control for individuals with type 1 diabetes mellitus (T1DM). However, designing an effective control policy for the AP remains challenging due to the complex physiological processes, delayed insulin response, and inaccurate glucose measurements. While model predictive control (MPC) offers safety and stability through the dynamic model and safety constraints, it lacks individualization and is adversely affected by unannounced meals. Conversely, deep reinforcement learning (DRL) provides personalized and adaptive strategies but faces challenges with distribution shifts and substantial data requirements. We propose a hybrid control policy for the artificial pancreas (HyCPAP) to address the above challenges. HyCPAP combines an MPC policy with an ensemble DRL policy, leveraging the strengths of both policies while compensating for their respective limitations. To facilitate faster deployment of AP systems in real-world settings, we further incorporate meta-learning techniques into HyCPAP, leveraging previous experience and patient-shared knowledge to enable fast adaptation to new patients with limited available data. We conduct extensive experiments using the FDA-accepted UVA/Padova T1DM simulator across three scenarios. Our approaches achieve the highest percentage of time spent in the desired euglycemic range and the lowest occurrences of hypoglycemia. The results clearly demonstrate the superiority of our methods for closed-loop glucose management in individuals with T1DM. The study presents novel control policies for AP systems, affirming the great potential of proposed methods for efficient closed-loop glucose control.

Reinforcement Learning-Based Event-Triggered Active-Battery-Cell-Balancing Control for Electric Vehicle Range Extension

Optimal control techniques such as model predictive control (MPC) have been widely studied and successfully applied across a diverse field of applications. However, the large computational requirements for these methods result in a significant challenge for embedded applications. While event-triggered MPC (eMPC) is one solution that could address this issue by taking advantage of the prediction horizon, one obstacle that arises with this approach is that the event-trigger policy is complex to design to fulfill both throughput and control performance requirements. To address this challenge, this paper proposes to design the event trigger by training a deep Q-network reinforcement learning agent (RLeMPC) to learn the optimal event-trigger policy. This control technique was applied to an active-cell-balancing controller for the range extension of an electric vehicle battery. Simulation results with MPC, eMPC, and RLeMPC control policies are presented along with a discussion of the challenges of implementing RLeMPC.

A neural state-space-based model predictive technique for effective vibration control in nano-beams

Model predictive control (MPC) is a cutting-edge control technique, but its susceptibility to inaccuracies in the model remains a challenge for embedded systems. In this study, we propose a data-driven MPC framework to address this issue and achieve robust and adaptable performance. Our framework involves systematically identifying system dynamics and learning the MPC policy through function approximations. Specifically, we introduce a system identification method based on the Deep neural network (DNN) and integrate it with MPC. The function approximation capability of DNN enables the controller to learn the nonlinear dynamics of the system then the MPC policy is established based on the identified model. Also, through an added control term the robustness and convergence of the closed-loop system are guaranteed. Then the governing equation of a non-local strain gradient (NSG) nano-beam is presented. Finally, the proposed control scheme is used for vibration suppression in the NSG nano-beam. To validate the effectiveness of our approach, the controller is applied to the unknown system, meaning that solely during the training phase of the neural state-space-based model we relied on the data extracted from the time history of the beam’s deflection. The simulation results conclusively demonstrate the remarkable performance of our proposed approach in effectively suppressing vibrations.

Model predictive control policy design, solutions, and stability analysis for longitudinal vehicle control considering shockwave damping

Longitudinal vehicle control models, such as Adaptive Cruise Control (ACC) and Cooperative Adaptive Cruise Control (CACC), have been the core component of many Automated Driving Systems (ADS) or Advanced Driver Assistance System (ADAS). ACC and CACC systems make vehicles drive with faster reaction time and smaller following distance than manual vehicles, alleviating the congestions and attenuating traffic disturbances. Many existing ACC and CACC control models focus on three strategies that can indirectly mitigate congestions and damp traffic shockwaves: time headway suppression, stability improvement, and adaptive mode-switching control based on traffic conditions. Although congestion mitigation and shockwave damping are achievable with those control strategies, the improvement is often limited to the by-product effect of their main control objectives of headway suppression and system stability. Many adaptive models also need to calibrate many parameters for different road conditions.This paper proposes a new ACC and CACC framework that directly integrates the congestion shockwave damping and traffic congestion mitigation into the objective function of a dynamic control framework. A Model Predictive Control (MPC) based dynamic optimization framework is proposed to balance the trade-offs among multiple objectives, including safety, efficiency, shockwave, elasticity, and driver’s comfort. The proposed framework is applied in both ACC and CACC platoons mixed with manual vehicles, and five different control models are explored. The stability conditions of the proposed models are derived analytically and analyzed numerically. Three experimental studies, including platoon, ring-road, and traffic simulation studies, indicate promising results of proposed models in reducing shockwave propagation speed and mitigating traffic congestion under different environments compared with existing ACC and CACC models.

An Improved Data Augmentation Scheme for Model Predictive Control Policy Approximation

This paper considers the problem of data generation for MPC policy approximation. Learning an approximate MPC policy from expert demonstrations requires a large data set consisting of optimal state-action pairs, sampled across the feasible state space. Yet, the key challenge of efficiently generating the training samples has not been studied widely. Recently, a sensitivity-based data augmentation framework for MPC policy approximation was proposed, where the parametric sensitivities are exploited to cheaply generate several additional samples from a single offline MPC computation. The error due to augmenting the training data set with inexact samples was shown to increase with the size of the neighborhood around each sample used for data augmentation. Building upon this work, this letter paper presents an improved data augmentation scheme based on predictor-corrector steps that enforces a user-defined level of accuracy, and shows that the error bound of the augmented samples are independent of the size of the neighborhood used for data augmentation.

Designing a resilient supply chain through a robust adaptive model predictive control policy under perishable goods and uncertain forecast information

We consider the inventory control problem for Supply Chains (SC) with deteriorating items and an uncertain future customer demand freely varying inside a given compact set. The problem is to define a resilient Replenishment Policy (RP) keeping the actual inventory level as close as possible to a desired reference trajectory. This requirement should be satisfied despite uncertainties on the decay factor of stocked goods and unexpected customer demand patterns outside the bounds of the compact set. We propose a method based on a Resilient Robust Model Predictive Control (RRMPC) approach. This requires dealing with a Min-Max Constrained Optimization Problem (MMCOP). To reduce the numerical complexity of the algorithm, the control signal is parametrized using B-spline functions.

A Sensitivity-Based Data Augmentation Framework for Model Predictive Control Policy Approximation

Approximating model predictive control (MPC) policy using expert-based supervised learning techniques requires labeled training datasets sampled from the MPC policy. This is typically obtained by sampling the feasible state space and evaluating the control law by solving the numerical optimization problem offline for each sample. Although the resulting approximate policy can be cheaply evaluated online, generating large training samples to learn the MPC policy can be time-consuming and prohibitively expensive. This is one of the fundamental bottlenecks that limit the design and implementation of MPC policy approximation. This technical article aims to address this challenge, and proposes a novel sensitivity-based data augmentation scheme for direct policy approximation. The proposed approach is based on exploiting the parametric sensitivities to cheaply generate additional training samples in the neighborhood of the existing samples.

Learning Stochastic Parametric Diferentiable Predictive Control Policies

The problem of synthesizing stochastic explicit model predictive control policies is known to be quickly intractable even for systems of modest complexity when using classical control-theoretic methods. To address this challenge, we present a scalable alternative called stochastic parametric differentiable predictive control (SP-DPC) for unsupervised learning of neural control policies governing stochastic linear systems subject to nonlinear chance constraints. SP-DPC is formulated as a deterministic approximation to the stochastic parametric constrained optimal control problem. This formulation allows us to directly compute the policy gradients via automatic differentiation of the problem's value function, evaluated over sampled parameters and uncertainties. In particular, the computed expectation of the SP-DPC problem's value function is back propagated through the closed-loop system rollouts parametrized by a known nominal system dynamics model and neural control policy which allows for direct model-based policy optimization. We demonstrate the computational efficiency and scalability of the proposed policy optimization algorithm in three numerical examples, including systems with a large number of states or subject to nonlinear constraints.

Multi-Rate Control Design Under Input Constraints via Fixed-Time Barrier Functions

In this paper, we introduce the notion of periodic safety, which requires that the system trajectories periodically visit a subset of a forward-invariant safe set, and utilize it in a multi-rate framework where a high-level planner generates a reference trajectory that is tracked by a low-level controller under input constraints. We introduce the notion of fixed-time barrier functions which is leveraged by the proposed low-level controller in a quadratic programming framework. Then, we design a model predictive control policy for high-level planning with a bound on the rate of change for the reference trajectory to guarantee that periodic safety is achieved. We demonstrate the effectiveness of the proposed strategy on a simulation example, where the proposed fixed-time stabilizing low-level controller shows successful satisfaction of control objectives, whereas an exponentially stabilizing low-level controller fails.

Generalized reinforcement learning for building control using Behavioral Cloning

Advanced building control methods such as model predictive control (MPC) offer significant benefits to both consumers and grid operators, but high computational requirements have acted as barriers to more widespread adoption. Local control computation requires installation of expensive computational hardware, while cloud computing introduces data security and privacy concerns. In this paper, we drastically reduce the local computational requirements of advanced building control through a reinforcement learning (RL)-based approach called Behavioral Cloning, which represents the MPC policy as a neural network that can be locally implemented and quickly computed on a low-cost programmable logic controller. While previous RL and approximate MPC methods must be specifically trained for each building, our key improvement is that the proposed controller can generalize to many buildings, electricity rates, and thermostat setpoint schedules without additional, effort-intensive retraining. To provide this versatility, we have adapted the traditional Behavioral Cloning approach through two innovations: (1) a constraint-informed parameter grouping (CIPG) method that provides a more efficient representation of the training data and (2) a new deep learning model-structure called reverse-time recurrent neural networks (RT-RNN) that allows future information to flow backward in time to more effectively interpret the temporal information in disturbance predictions. The result is an easy-to-deploy, generalized behavioral clone of MPC that can be implemented on a programmable logic controller and requires little building-specific controller tuning, reducing the effort and costs associated with implementing smart residential heat pump control.

Gnu-RL: A Practical and Scalable Reinforcement Learning Solution for Building HVAC Control Using a Differentiable MPC Policy

We propose Gnu-RL: a novel approach that enables real-world deployment of reinforcement learning (RL) for building control and requires no prior information other than historical data from existing Heating Ventilation and Air Conditioning (HVAC) controllers. In contrast with existing RL agents, which are opaque to expert interrogation and need ample training to achieve reasonable control performance, Gnu-RL is much more attractive for real-world applications. Furthermore, Gnu-RL avoids the need to develop and calibrate simulation environments for each building or system under control, thus making it highly scalable. To achieve this, we bootstrap the Gnu-RL agent with domain knowledge and expert demonstration. Specifically, Gnu-RL adopts a recently-developed Differentiable Model Predictive Control (MPC) policy, which encodes domain knowledge on planning and system dynamics, making it both sample-efficient and interpretable. Prior to any interaction with the environment, a Gnu-RL agent is pre-trained on historical data using imitation learning, enabling it to match the behavior of the existing controller. Once it is put in charge of controlling the environment, the agent continues to improve its policy end-to-end, using a policy gradient algorithm. We evaluate Gnu-RL in both simulation studies and a real-world testbed. Firstly, we validated the Gnu-RL agent is indeed practical and scalable by applying it to three commercial reference buildings in simulation, and demonstrated the superiority of the Differentiable MPC policy compared to a generic neural network. In another simulation experiment, our approach saved 6.6% of the total energy compared to the best published RL result for the same environment, while maintaining a higher level of occupant comfort. Finally, Gnu-RL was deployed to control the airflow to a real-world conference room with a Variable-Air-Volume (VAV) system during a three-week period. Our results show that Gnu-RL reduced cooling demand by 16.7% compared to the existing controller and tracked the temperature set-point better.

Robust model predictive control for nonlinear parameter varying systems without computational delay

SummaryThis article proposes a one‐step ahead robust model predictive control (MPC) for discrete‐time Lipschitz nonlinear parameter varying (NLPV) systems subject to disturbances. Within the proposed design framework, the optimization that generates the MPC policy to be implemented at next time instant is executed in advance during the current sampling period based on future state prediction. This new feature allows avoidance of the online computational delay existing in the traditional MPC settings and improves the control performance. The proposed MPC is proved to be recursively feasible with the guarantee for robust closed‐loop system stability and satisfaction of input and output constraints. A tractable linear matrix inequality (LMI) optimization problem is formulated to compute the control gains at each time instant. The computational complexity of the obtained LMI problem is also analyzed. The one‐step ahead robust MPC is further developed to cover discrete‐time Lipschitz NLPV systems with disturbance compensation. Efficacy and performance improvement of the design are demonstrated through a numerical example and an application to adaptive cooperative cruise control for automated vehicles under variable road geometry.

Electric Satellite Station Keeping, Attitude Control, and Momentum Management by MPC

We propose a model predictive control (MPC) policy for simultaneous station keeping, attitude control, and momentum management of a low-thrust nadir-pointing geostationary satellite equipped with reaction wheels and ON–OFF electric thrusters mounted on boom assemblies. Attitude control is performed using an inner loop SO(3)-based control law with the reaction wheels, while the outer loop MPC policy maintains the satellite within a narrow station-keeping window and performs momentum management using electric thrusters. For reducing propellant consumption, our MPC uses two different prediction horizons: a short horizon for the states associated with the orbit’s inclination and a longer horizon for all other states. Furthermore, to handle the ON–OFF nature of the thruster while retaining low computational burden, we develop a strategy for quantizing the continuous thrust command, which also allows for trading off the number thrust pulses and fuel consumption. We validate the controller in a closed-loop simulation with the high-precision orbit propagation provided by the Systems Tool Kit (STK) and assess the robustness to model uncertainty and measurement noise.

Multi-Rate Control Design Leveraging Control Barrier Functions and Model Predictive Control Policies

In this letter we present a multi-rate control architecture for safety critical systems. We consider a high level planner and a low level controller which operate at different frequencies. This multi-rate behavior is described by a piecewise nonlinear model which evolves on a continuous and a discrete level. First, we present sufficient conditions which guarantee recursive constraint satisfaction for the closed-loop system. Afterwards, we propose a control design methodology which leverages Control Barrier Functions (CBFs) for low level control and Model Predictive Control (MPC) policies for high level planning. The control barrier function is designed using the full nonlinear dynamical model and the MPC is based on a simplified planning model. When the nonlinear system is control affine and the high level planning model is linear, the control actions are computed by solving convex optimization problems at each level of the hierarchy. Finally, we show the effectiveness of the proposed strategy on a simulation example, where the low level control action is updated at a higher frequency than the high level command.

Design and Control of Park & Charge Lanes for Carsharing Services with Highly-Automated Electric Vehicles

Carsharing operators could benefit from vehicle automation even before full vehicle automation is available city-wide: so-called Park- & Charge Lanes (PCL) installed in closed environments can increase customer convenience and reduce costs for parking space and charging operations. The concept introduced in this study comprises the stacking of vehicles in several lanes of optimized width, the division of lanes into charging and parking areas, and control strategies for efficient operation. Compared to conventional parking lots with two lanes and two perpendicularly arranged parking spaces, the stacking of vehicles allows for space reductions of up to 43 %. Additional cost savings can be achieved, since it is not necessary to equip every parking space with an inductive charging plate. Splitting each lane into a parking and a charging area makes the optimal control problem non-trivial: In order to provide the vehicles with a battery level that is high enough to serve customer requests, the PCL has to be controlled in a smart way. Both rule-based and model-predictive control policies are developed for assigning arriving vehicles to lanes and selecting vehicles for customer requests. An event-based simulation framework is created in order to test the performance of the introduced policies for the resulting dynamic and stochastic PCL problem. The best of the four described rule-based policies performs nearly as good as the implemented model-predictive control approach in the numerical experiment. The model-predictive control policy outperforms the random lane selection by 27 %, which clearly refects the benefit of using advanced control strategies.

Low-Latency Robust MPC for CACC under Variable Road Geometry

Cooperative adaptive cruise control (CACC) has attracted much research attention, due to its great potential in improving traffic throughput, safety and energy efficiency. This paper aims to address the following problems that are rarely investigated in the literature: (i) the time delay caused by online computation of the optimal control action, and (ii) robustly stable CACC under variable road geometry. To this end, a one-step ahead robust model predictive control (MPC) is developed for achieving CACC and lane keeping (LK) of the followers in the platoon, by leveraging vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communications. In the proposed design, the current MPC policy is generated one-step ahead during the previous sampling period to avoids the optimization-induced time delay existing in the traditional MPC. LK control is incorporated with CACC to ensure vehicle lateral stability and vehicle following under variable road geometry. The MPC design is formulated as an easily solved linear matrix inequality (LMI) optimization problem with consideration of the control input limits and constraints on platooning errors and lateral displacement. Effectiveness of the proposed MPC and its advantages over the traditional MPC are verified by simulating a vehicle platoon on roads with time-varying bank, curvature and grade.

Real-time optimization of the integrated gas and power systems using hybrid approximate dynamic programming

This paper proposes a hybrid approximate dynamic programming (HADP) approach for the optimal operation of integrated gas and power systems (IGPS) under the stochastic environment. The proposed HADP, combining the advantages of the model predictive control (MPC) and approximate dynamic programming (ADP), decomposes the multi-time-period optimization into multiple sequential subproblems by solving Bellman’s equation forward through time. Historical data is utilized to build the approximate value functions so that the influence of current decisions on the future is estimated. And hence, the HADP algorithm can obtain a near-optimal solution through the whole time horizon of interest. Meanwhile, the MPC policy is embedded in the HADP to replace the long-term forecast with short-term or even real-time prediction. This further improves the optimality of the decisions made by the proposed HADP. The simulation results on the IGPS demonstrate the proposed HADP outperforms alternative solutions.

Station keeping and momentum management of low-thrust satellites using MPC

This work proposes a Model Predictive Control (MPC) policy for simultaneous station keeping and momentum management of a low-thrust nadir-pointing satellite in geostationary orbit around the Earth. The satellite is equipped with six electrically powered thrusters and three axisymmetric reaction wheels, which must be coordinated to control the satellite's orbital position and, concurrently, unload the wheels' stored angular momentum. The MPC policy enforces constraints that maintain the satellite in a tight latitude and longitude window and in a tight nadir-pointing attitude configuration, while minimizing the delta-v provided by the thrusters. The MPC policy exploits a prediction model of the environmental disturbance forces in order to significantly reduce the delta-v required for station keeping, and enforces constraints determined by the thruster configuration to select control forces and torques that can be generated by the propulsion system. Numerical simulations of the control policy in closed-loop with the satellite nonlinear dynamics under high-precision orbit propagation provided by Systems Tool Kit (STK) that validate the performance of the proposed design in terms of thruster usage and constraint enforcement are presented.

MPC policy learning using DNN for human following control without collision

Model predictive control has recently been applied to a wide variety of motion control systems. Model predictive control can be used to generate optimized control inputs with excellent performance considering inequality constraints to the control inputs, control outputs, and state variables. However, the computational load for this method is too heavy for implementation in most actual systems because the quadratic programming problem must be solved within the sampling period. As the number of inequality constraints, control variables, and state variables in the control system increases, more calculation time is required. In this study, a deep neural network designed to learn the model predictive control policy was developed to reduce the computational load. It is expected that a relatively small neural network can be used to learn the model predictive control policy. In the proposed system, the motion controller calculates the learned neural network in real time instead of solving the quadratic programming problem, realizing almost the same control performance as the original model predictive control approach. The effectiveness of the proposed approach was verified by applying it to the control of a personal robot designed to follow the user, which can provide daily support to the elderly. In Matlab simulations, the calculation time for the proposed approach was approximately times faster than that of the conventional method of solving the quadratic programming problem. In addition, an experiment using an actual personal robot was conducted to confirm the control performance.

An application of economic model predictive control to inventory management in hospitals

In this paper, we present experimental results from the application of model predictive control (MPC) to inventory management in a real hospital. In particular, the stock levels of ten different drugs that belong to the same laboratory have been controlled by using an MPC policy. The results obtained after four months show that the adopted approach outperforms the method employed by the hospital and reduces both the average stock levels and the work burden of the pharmacy department. This paper also paper presents some practical insights regarding the application of advanced control methods in this context.