Guarantee Constraint Satisfaction Research Articles

A Distributed Model Predictive Control Strategy for Constrained Multi-Agent Systems: The Uncertain Target Capturing Scenario

In this paper, a distributed control architecture is presented for addressing the target capturing problem of a multi-agent system whose dynamics is described by double integrator models subject to bounded disturbance effects. Starting from novel kinematic models used as reference trajectories, the aim consists in driving the multi-agent system within the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">containment</i> region without entering the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">distancing</i> one and, whenever necessary, there remaining confined. The underlying control problem has been tackled by means of model predictive control arguments. In particular, two different distributed strategies have been developed, and adequately switched to each other, in order to guarantee constraint satisfaction within the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">capturing</i> region despite any disturbance realization. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This paper proposes a methodological solution for dealing with the capturing problem for multi-agent systems operating in uncertain environments where the target is defined by the ring resulting from the distancing and containment ellipsoids. Differently from the existing literature, the proposed approach combines into a unique framework properties coming from potential fields theory and distributed model predictive control philosophy. It is interesting to put in light that the underlying control strategy allows the users to face surveillance and rescue operations, to cite a few, in computational affordable way due to the required memory resources because most of computations can be straightforwardly moved in the off-line phase.

Pure-Past Action Masking

We present Pure-Past Action Masking (PPAM), a lightweight approach to action masking for safe reinforcement learning. In PPAM, actions are disallowed (“masked”) according to specifications expressed in Pure-Past Linear Temporal Logic (PPLTL). PPAM can enforce non-Markovian constraints, i.e., constraints based on the history of the system, rather than just the current state of the (possibly hidden) MDP. The features used in the safety constraint need not be the same as those used by the learning agent, allowing a clear separation of concerns between the safety constraints and reward specifications of the (learning) agent. We prove formally that an agent trained with PPAM can learn any optimal policy that satisfies the safety constraints, and that they are as expressive as shields, another approach to enforce non-Markovian constraints in RL. Finally, we provide empirical results showing how PPAM can guarantee constraint satisfaction in practice.

ConSequence: Synthesizing Logically Constrained Sequences for Electronic Health Record Generation

Generative models can produce synthetic patient records for analytical tasks when real data is unavailable or limited. However, current methods struggle with adhering to domain-specific knowledge and removing invalid data. We present ConSequence, an effective approach to integrating domain knowledge into sequential generative neural network outputs. Our rule-based formulation includes temporal aggregation and antecedent evaluation modules, ensured by an efficient matrix multiplication formulation, to satisfy hard and soft logical constraints across time steps. Existing constraint methods often fail to guarantee constraint satisfaction, lack the ability to handle temporal constraints, and hinder the learning and computational efficiency of the model. In contrast, our approach efficiently handles all types of constraints with guaranteed logical coherence. We demonstrate ConSequence's effectiveness in generating electronic health records, outperforming competitors in achieving complete temporal and spatial constraint satisfaction without compromising runtime performance or generative quality. Specifically, ConSequence successfully prevents all rule violations while improving the model quality in reducing its test perplexity by 5% and incurring less than a 13% slowdown in generation speed compared to an unconstrained model.

CVaR-Constrained Policy Optimization for Safe Reinforcement Learning.

Current constrained reinforcement learning (RL) methods guarantee constraint satisfaction only in expectation, which is inadequate for safety-critical decision problems. Since a constraint satisfied in expectation remains a high probability of exceeding the cost threshold, solving constrained RL problems with high probabilities of satisfaction is critical for RL safety. In this work, we consider the safety criterion as a constraint on the conditional value-at-risk (CVaR) of cumulative costs, and propose the CVaR-constrained policy optimization algorithm (CVaR-CPO) to maximize the expected return while ensuring agents pay attention to the upper tail of constraint costs. According to the bound on the CVaR-related performance between two policies, we first reformulate the CVaR-constrained problem in augmented state space using the state extension procedure and the trust-region method. CVaR-CPO then derives the optimal update policy by applying the Lagrangian method to the constrained optimization problem. In addition, CVaR-CPO utilizes the distribution of constraint costs to provide an efficient quantile-based estimation of the CVaR-related value function. We conduct experiments on constrained control tasks to show that the proposed method can produce behaviors that satisfy safety constraints, and achieve comparable performance to most safe RL (SRL) methods.

Constrained continuous-action reinforcement learning for supply chain inventory management

Reinforcement learning (RL) is a promising solution for difficult decision-making problems, such as inventory management in chemical supply chains. However, enabling RL to explicitly consider known environment constraints is crucial for safe deployment in practical applications. This work incorporates recent tools for optimization over trained neural networks to introduce two algorithms for safe training and deployment of RL, with a focus on supply chains. Specifically, we use optimization over trained neural-network state–action value functions (i.e., a critic function) to directly incorporate constraints when computing actions in a continuous action space. Furthermore, we introduce a second algorithm that guarantees constraint satisfaction during deployment by directly implementing actions from constrained optimization of a trained value function. The algorithms are compared against state-of-the-art algorithms TRPO, CPO, and RCPO using a computational supply chain case study.

Robust MPC-RG for an autonomous racing vehicle considering obstacles and the battery state of charge

The design of a controller able to deal with uncertainties and physical constraints plays an essential role in fast and complex systems. Then, a reference governor approach based on model predictive control (MPC-RG) for an autonomous racing vehicle is proposed. The MPC-RG guarantees constraint satisfaction and recursive feasibility online while including obstacle avoidance capability and energy-aware management by solving a multi-objective optimization problem. In particular, a trade-off between maximizing the longitudinal velocity and the state of charge of the vehicle’s battery, as well as minimizing the variation of control actions is adopted. Moreover, the proposed MPC-RG is combined with a state-feedback linear quadratic regulator (LQR) and a Kalman filter (KF) to compensate for modeling errors and exogenous disturbances, as well as to estimate the unmeasured lateral velocity. In fact, for control and estimation purposes, a data-driven Takagi–Sugeno (TS) model trained by an adaptive neuro-fuzzy inference network is used. The performance of the developed approach is assessed in simulations using a well-known case study based on a 1/10 scale RC electric car.

Feedback control to guarantee marking constraints in timed event graphs including disturbances: Application to disassembly systems

This paper addresses the control problem of Timed Event Graphs (TEGs) with disturbance transitions, which are subject to marking constraints. In these graphical models, disturbances are uncontrollable input transitions and constraints correspond to the maximum limits of tokens in some places which must not be exceeded. An analytical methodology based on the use of Min-Plus formalisms to synthesize feedback control laws that guarantee the concerned constraints is established. For this, Min-Plus linear models describe the dynamic behavior of the TEGs, and marking constraints are represented by inequalities in Min-Plus algebra. Sufficient conditions are defined to verify the existence of feedback control laws guaranteeing constraint satisfaction in disturbed TEGs. To verify the effectiveness of these proposed theoretical approaches, a real case study of a disturbed disassembly system with limited stocks of components is processed.

Power Control With QoS Guarantees: A Differentiable Projection-Based Unsupervised Learning Framework

Deep neural networks (DNNs) are emerging as a potential solution to solve NP-hard wireless resource allocation problems. However, in the presence of intricate constraints, e.g., users’ quality-of-service (QoS) constraints, guaranteeing constraint satisfaction becomes a fundamental challenge. In this paper, we propose a novel unsupervised learning framework to solve the classical power control problem in a multi-user interference channel, where the objective is to maximize the network sum-rate under users’ minimum data rate or QoS requirements and power budget constraints. Utilizing a differentiable projection function, two novel deep learning (DL) solutions are pursued. The first is called Deep Implicit Projection Network (DIPNet), and the second is called Deep Explicit Projection Network (DEPNet). DIPNet utilizes a differentiable convex optimization layer to implicitly define a projection function. On the other hand, DEPNet uses an explicitly-defined projection function, which has an iterative nature and relies on a differentiable correction process. DIPNet requires convex constraints; whereas, the DEPNet does not require convexity and has a reduced computational complexity. To enhance the sum-rate performance of the proposed models even further, Frank-Wolfe algorithm (FW) has been applied to the output of the proposed models. Extensive simulations depict that the proposed DNN solutions not only improve the achievable data rate but also achieve zero constraint violation probability, compared to the existing DNNs. The proposed solutions outperform the classic optimization methods in terms of computation time complexity.

A predictive chance constraint rebalancing approach to mobility-on-demand services

This paper considers the problem of supply-demand imbalances in Mobility-on-Demand (MoD) services. These imbalances occur due to uneven stochastic travel demand and can be mitigated by proactively rebalancing empty vehicles to areas where the demand is high. To achieve this, we propose a method that takes into account uncertainties of predicted travel demand while minimizing pick-up time and rebalance mileage for autonomous MoD ride-hailing. More precisely, first travel demand is predicted using Gaussian Process Regression (GPR) which provides uncertainty bounds on the prediction. We then formulate a stochastic model predictive control (MPC) for the autonomous ride-hailing service and integrate the demand predictions with uncertainty bounds. In order to guarantee constraint satisfaction in the optimization under estimated stochastic demand prediction, we employ a probabilistic constraining method with user-defined confidence interval, using Chance Constrained MPC (CCMPC). The benefits of the proposed method are twofold. First, travel demand uncertainty prediction from data can naturally be embedded into the MoD optimization framework, allowing us to keep the imbalance at each station below a certain threshold with a user-defined probability. Second, CCMPC can be relaxed into a Mixed-Integer-Linear-Program (MILP) and the MILP can be solved as a corresponding Linear-Program, which always admits an integral solution. Our transportation simulations show that by tuning the confidence bound on the chance constraint, close to optimal oracle performance can be achieved, with a median customer wait time reduction of 4% compared to using only the mean prediction of the GPR.

Inaccuracy Matters: Accounting for Solution Accuracy in Event-Triggered Nonlinear Model Predictive Control

We consider the effect of using approximate system predictions in event-triggered control schemes. These approximations often result from using numerical transcription methods for solving continuous-time optimal control problems. Mesh refinement can guarantee upper bounds on the error in the differential equations that model the system dynamics. We employ the accuracy guarantees of a mesh refinement scheme to show that the proposed event-triggering scheme, which compares the measured system with approximate state predictions, can be used with a guaranteed strictly positive inter-update time. Furthermore, if knowledge of the employed transcription scheme or the approximation errors are available, then better online estimates of inter-update times can be obtained. We also detail a method of tightening constraints on the approximate system trajectory to guarantee constraint satisfaction of the continuous-time system. This is the first work to incorporate prediction accuracy in triggering metrics to guarantee reliable lower bounds for inter-update times and perform solution-dependent constraint tightening.

Data-driven predictive control in a stochastic setting: a unified framework

Data-driven predictive control (DDPC) has been recently proposed as an effective alternative to traditional model-predictive control (MPC) for its unique features of being time-efficient and unbiased with respect to the oracle solution. Nonetheless, it has also been observed that noise may strongly jeopardize the final closed-loop performance, since it affects both the data-based system representation and the control update computed from the online measurements. Recent studies have shown that regularization is potentially a successful tool to counteract the effect of noise. At the same time, regularization requires the tuning of a set of penalty terms, whose choice might be practically difficult without closed-loop experiments In this paper, by means of subspace identification tools, we pursue a three-fold goal: (i) we set up a unified framework for the existing regularized data-driven predictive control schemes for stochastic systems; (ii) we introduce γ-DDPC, an efficient two-stage scheme that splits the optimization problem in two parts: fitting the initial conditions and optimizing the future performance, while guaranteeing constraint satisfaction; (iii) we discuss the role of regularization for data-driven predictive control, providing new insight on when and how it should be applied. A benchmark numerical case study finally illustrates the performance of γ-DDPC, showing how controller design can be simplified in terms of tuning effort and computational complexity when benefiting from the insights coming from the subspace identification realm.

LMI-based Data-Driven Robust Model Predictive Control

Predictive control, which is widely used currently, bases on a model of the system to compute the applied input optimizing the future system behavior. If the nominal models are not given or are uncertain, data-driven model predictive control approaches can be employed, where control input is directly obtained from past measured trajectories. Using a data informativity framework and Finsler's lemma, we propose a data-driven robust linear matrix inequality-based model predictive control scheme that considers input and state constraints for linear parameter-varying systems and Lur'e-type nonlinear systems. Using these data, we formulate the problem as a semi-definite optimization problem, whose solution provides the matrix gain for the linear feedback, while the decisive variables are independent of the length of the measurement data. The designed controller stabilizes the closed-loop system asymptotically and guarantees constraint satisfaction. Numerical examples are conducted to illustrate the method.

Multi-Agent Reinforcement Learning with Information-sharing Constrained Policy Optimization for Global Cost Environment

Multi-agent Reinforcement Learning (MARL) is a machine learning method that solves problems by using multiple learning agents in a data-driven manner. Because of the advantage of utilizing multiple agents simultaneously, MARL has become an efficient solution to large-scale problems in a wide range of fields. However, as with general single-agent reinforcement learning, MARL requires trial and error to acquire the appropriate policies for each agent in the learning process. Therefore, how to guarantee performance and constraint satisfaction in MARL is a critical issue for application to real-world problems. In this study, we propose an Information-sharing Constrained Policy Optimization (IsCPO) method for MARL that guarantees constraint satisfaction during learning. In detail, IsCPO sequentially updates the policies of multiple agents in random order while sharing information of the surrogate costs and KL-divergence for evaluating the current and updated policies to the next agent. In addition, if there are no candidates of policies to be updated in accordance with the shared information, IsCPO skips updating the policies of the rest of the agents until the next iteration. As a result, IsCPO makes it possible to acquire the individual suboptimal policies of agents, satisfying constraints on global costs related to the state of the environment and the actions from multiple agents. We also introduce a practical algorithm for IsCPO that simplifies its implementation by adopting several mathematical approximations. Finally, we show the validity and effectiveness through simulation results on a multiple cart-pole problem and base station sleep control problem in a mobile network.

Robust Tube-Based Model Predictive Control for Wave Energy Converters

This paper proposes an efficient robust tube-based model predictive control (RTMPC) strategy for energy-maximization control of wave energy converters (WECs) subject to constraints due to safety considerations. Compared with the existing MPC strategies developed for the WEC control problem, the RTMPC method provides an effective approach to explicitly handle plant-model mismatches with guaranteed constraint satisfaction, contributing to improved energy capture efficiency. The fundamental idea is to integrate disturbance invariant sets into the MPC scheme for energy-maximization control to form a tube-based predictive controller, which enhances the robustness of MPC for a WEC without increasing online computational complexity. The resulting RTMPC controller can bound the WEC plant trajectories in a tube centered around a nominal WEC model trajectory, and uncertainties from un-modeled WEC dynamics and unmeasured disturbances can be mitigated by an error feedback portion. Numerical simulations demonstrate the effectiveness of the proposed control strategy.

On Tuning Parameterized Control Policies Online for Safety-Critical Systems -Applied to Biomedical Systems

There are different approaches to tune a control policy that would result in a desired closed-loop performance. The typical design flow involves tuning the control policies offline using (high-fidelity) simulators until satisfactory performance is achieved. This paper on the other hand, considers the problem of tuning parameterized control policies directly by interacting with the real system that also has safety-critical constraints. We use safe Bayesian optimization using interior-point methods to tune the parameterized control policy online that guarantees constraint satisfaction with high probability. The proposed framework is applied to a personalized artificial pancreas system for type 1 diabetes. The paper shows that the parameterized control policy used for blood glucose regulation can be safely tuned to personalize the controller for each individual patient using our approach and thus improve its performance.

Robust model predictive control for linear sampled-data systems with irregular sampling times

This paper presents a sampled-data tube-based robust MPC scheme for linear continuous-time systems with irregular sampling times. The sampled-data control law is updated only at discrete sampling instances, but the proposed controller guarantees constraint satisfaction of the continuous-time state for all times. The proposed MPC scheme allows for two sources of uncertainty, (i) uncertain sampling times and (ii) an additional disturbance to the continuous-time system. A constraint adaptation is presented to handle this setting in a rigid tube MPC framework. Constraint satisfaction and convergence of the continuous-time state are shown for the proposed MPC scheme.

Model Predictive Control of an Industrial Distillation Column Based on a Hybrid Model: Adapting the Domain of Validity

In any model predictive control (MPC) implementation, the key factor is the quality of the employed system model in order to guarantee constraint satisfaction and to optimize the performance of the controlled system. However, it is difficult in practice to model all the phenomena of the true system which leads to plant-model mismatch. We propose to augment a simple first principles based model with a machine learning model to capture the plant dynamics which are not represented in the semi-rigorous model. The use of data-based models however can be problematic outside the region where the model was trained. In this work, we estimate the domain of validity of the data-based model using a one-class support vector machine (SVM) which is trained on a low-dimensional projection of the available training data. During the application of the controller, data of newly visited regions is collected and the domain of validity is adapted based on checking a performance criterion. The hybrid model incorporates a weighted contribution of the data-based model based on its predicted validity. We demonstrate the versatility and the benefits of the proposed adaptive MPC based on a hybrid model (AHMPC) approach via an industrial distillation column case study.

Receding-Horizon Control of Constrained Switched Systems with Neural Networks as Parametric Function Approximators

This work studies receding-horizon control of discrete-time switched linear systems subject to polytopic constraints for the continuous states and inputs. The objective is to approximate the optimal receding-horizon control strategy for cases in which the online computation is intractable due to the necessity of solving mixed-integer quadratic programs in each discrete time instant. The proposed approach builds upon an approximated optimal finite-horizon control law in closed-loop form with guaranteed constraint satisfaction. The paper derives the properties of recursive feasibility and asymptotic stability for the proposed approach. A numerical example is provided for illustration and evaluation of the approach.

Scheduling and Robust Invariance in Networked Control Systems

In Networked Control Systems (NCS), impairments of the communication channel can be disruptive to stability and performance. In this paper, we consider the problem of scheduling the access to limited communication resources for a number of decoupled systems subject to state and input constraints, whose loops need to be closed over the network. The schedule must be designed to robustly preserve the invariance property of each system, which in turn guarantees constraint satisfaction. To that end, we first focus on systematic offline scheduling design to preserve robust invariance, and afterwards on online scheduling design with the aim of improving performance and compensating for packet losses while guaranteeing recursive schedulability.

Robust distributed model predictive control of linear systems: Analysis and synthesis

To provide robustness of distributed model predictive control (DMPC), this work proposes a robust DMPC formulation for discrete-time linear systems subject to unknown-but-bounded disturbances. Taking advantage of the structure of certain classes of distributed systems seen in applications with inter-agent coupling, a novel robust DMPC is formulated. The proposed approach is characterised by separable terminal costs and locally robust terminal sets, with the latter sets adaptively estimated in the online optimisation problem. A constraint tightening approach based on a set-membership approach is used to guarantee constraint satisfaction for coupled subsystems in the presence of disturbances. Under this formulation, the closed-loop system is shown to be recursively feasible and input-to-state stable. To aid in the deployment of the proposed robust DMPC, a possible synthesis method and design conditions for practical implementation are presented. Finally, simulation results with a mass–spring–damper system are provided to demonstrate the proposed robust DMPC.