Risk-sensitive Control Research Articles

Risk sensitive control and an optimal reinsurance model of a network of insurance companies

Risk sensitive control and an optimal reinsurance model of a network of insurance companies

Lyapunov’s stability analysis for first degree polynomial systems, subject to risk-sensitive control

This work presents a novel methodology to verify the stability of first-degree stochastic polynomial systems under a Risk-Sensitive (RS) optimal control using Lyapunov’s stability analysis theory. The so-called Lyapunov’s indirect method is applied to prove its stability when it is combined with the dynamics of Riccati’s gain equation. The proposed methodology guarantees both the exponential stability of deterministic systems and the robustness of stochastic systems. Simulation results demonstrate the robustness, the effectiveness and the feasibility of this proposal.

Risk-sensitive control, single controller games and linear programming

Risk-sensitive control, single controller games and linear programming

Toward a Theoretical Foundation of Policy Optimization for Learning Control Policies

Gradient-based methods have been widely used for system design and optimization in diverse application domains. Recently, there has been a renewed interest in studying theoretical properties of these methods in the context of control and reinforcement learning. This article surveys some of the recent developments on policy optimization, a gradient-based iterative approach for feedback control synthesis that has been popularized by successes of reinforcement learning. We take an interdisciplinary perspective in our exposition that connects control theory, reinforcement learning, and large-scale optimization. We review a number of recently developed theoretical results on the optimization landscape, global convergence, and sample complexityof gradient-based methods for various continuous control problems, such as the linear quadratic regulator (LQR), [Formula: see text] control, risk-sensitive control, linear quadratic Gaussian (LQG) control, and output feedback synthesis. In conjunction with these optimization results, we also discuss how direct policy optimization handles stability and robustness concerns in learning-based control, two main desiderata in control engineering. We conclude the survey by pointing out several challenges and opportunities at the intersection of learning and control.

Duality between large deviation control and risk-sensitive control for Markov decision processes

Duality between large deviation control and risk-sensitive control for Markov decision processes

Adaptive power method for estimating large deviations in Markov chains.

We study the performance of a stochastic algorithm based on the power method that adaptively learns the large deviation functions characterizing the fluctuations of additive functionals of Markov processes, used in physics to model nonequilibrium systems. This algorithm was introduced in the context of risk-sensitive control of Markov chains and was recently adapted to diffusions evolving continuously in time. Here we provide an in-depth study of the convergence of this algorithm close to dynamical phase transitions, exploring the speed of convergence as a function of the learning rate and the effect of including transfer learning. We use as a test example the mean degree of a random walk on an Erdős-Rényi random graph, which shows a transition between high-degree trajectories of the random walk evolving in the bulk of the graph and low-degree trajectories evolving in dangling edges of the graph. The results show that the adaptive power method is efficient close to dynamical phase transitions, while having many advantages in terms of performance and complexity compared to other algorithms used to compute large deviation functions.

<i>G</i>-stochastic maximum principle for risk-sensitive control problem and its applications

<i>G</i>-stochastic maximum principle for risk-sensitive control problem and its applications

Risk-sensitive control for a class of diffusions with jumps

We consider a class of diffusions controlled through the drift and jump size, and driven by a jump Lévy process and a nondegenerate Wiener process, and we study infinite horizon (ergodic) risk-sensitive control problems for this model. We start with the controlled Dirichlet eigenvalue problem in smooth bounded domains, which also allows us to generalize current results in the literature on exit rate control problems. Then we consider the infinite horizon average risk-sensitive minimization and maximization problems on the whole domain. Under suitable hypotheses, we establish existence and uniqueness of a principal eigenfunction for the Hamilton–Jacobi–Bellman (HJB) operator on the whole space, and fully characterize stationary Markov optimal controls as the measurable selectors of this HJB equation.

Ergodic risk-sensitive control for regime-switching diffusions

In this article, we study the ergodic risk-sensitive control problem for controlled regime-switching diffusions. Under a blanket stability hypothesis, we solve the associated nonlinear eigenvalue problem for weakly coupled systems and characterize the optimal stationary Markov controls via a suitable verification theorem. We also consider the near-monotone case and obtain the existence of principal eigenfunction and optimal stationary Markov controls.

Indefinite risk-sensitive control

We consider a risk-sensitive control problem with indefinite cost matrices. All solutions to such an optimal control problem are obtained in an explicit closed-form by the completion of squares and the change of measure methods. Both the finite and infinite horizon cases are considered. As an application, we find all solutions to an optimal investment problem in a market with a stochastic interest rate.

CONTROL DE TEMPERATURA EN UN EVAPORADOR APLICANDO EL CONTROL RISK-SENSITIVE

This work designs a stochastic model of an evaporator of a refrigeration system, subject to specific conditions, applying the Risk-Sensitive (R-S) stochastic control equations with tracking, to control the evaporator temperature achieving great energy savings, where the actuator is the expansion valve (EEV).

Risk-sensitive discounted cost criterion for continuous-time Markov decision processes on a general state space

In this paper, we consider risk-sensitive discounted control problem for continuous-time jump Markov processes taking values in general state space. The transition rates of underlying continuous-time jump Markov processes and the cost rates are allowed to be unbounded. Under certain Lyapunov condition, we establish the existence and uniqueness of the solution to the Hamilton–Jacobi–Bellman equation. Also, we prove the existence of optimal risk-sensitive control in the class of Markov control and completely characterized the optimal control.

Risk-sensitive semi-Markov decision problems with discounted cost and general utilities

In this article we consider risk-sensitive control of semi-Markov processes with a discrete state space. We consider general utility functions and discounted cost in the optimization criteria. We consider random finite horizon and infinite horizon problems. Using a state augmentation technique we characterize the value functions and also prescribe optimal controls.

EFFECT OF RISK-SENSITIVE STOCHASTIC OPTIMAL CONTROL WITH TRACKING IN AN EVAPORATOR

This work presents an application of the Risk-Sensitive (R-S) control with tracking applied to a stochastic nonlinear system which models the operation of an electronic expansion valve (EEV) in a conventional evaporator. A novel dynamical stochastic equation represents the mathematical model of the evaporator system. The R-S stochastic optimal problem consists of the design of an optimal control u(t) such that the state reaches setpoint values (SP) and minimizes the exponential quadratic cost function. The presence of disturbances and errors in the sensor measurements is represented by Gauss white noise in the state equation, with the coefficient v(e/(2?^2 )) . One novel characteristic in this proposal is that the coefficient of the control into the state equation contains the state term. The error and exponential quadratic cost function show that the R-S control has a better performance versus the classical PID (Proportional, Integral Derivative) control. Key Words: Optimal Risk-Sensitive control with tracking, modelling of the evaporator.

Data-Driven Risk-Sensitive Control for Personalized Lane Change Maneuvers

Most current research in the field of autonomous vehicle control assumes that all vehicles will follow the same patterns of automated driving behavior, resulting in systems with “conservative” or “average” driving styles. These systems may not be acceptable to drivers who prefer a more aggressive style of driving, however, while extremely cautious drivers may consider the standard outputs to be too aggressive. To address this problem, in this paper, we introduce Risk Sensitive Control (RSC), an inverse optimal control algorithm that estimates risk-sensitive driving features and incorporates them into a receding-horizon controller. RSC uses a meta-learning algorithm to update the parameters of the cost function, continuously improving the controller online as more and more driving data is gathered from the user and subjective risk feedback. An estimator takes into account individual differences in subjective risk analysis, in terms of driving features and surrounding vehicle locations, by adjusting the cost function and its constraints. We test this approach using five lane change scenarios, some safe and some risky, with thirty real drivers in a CARLA simulation environment. Our quantitative and qualitative evaluations demonstrate that the proposed framework is able to generate a user’s preferred driving maneuvers during lane changes, i.e., control commands the user associates with lower subjective risk, outperforming conventional, model-based predictive control methods in terms of replicating the user’s own driving behavior.

Ergodic risk-sensitive control of Markov processes on countable state space revisited

We consider a large family of discrete and continuous time controlled Markov processes and study an ergodic risk-sensitive minimization problem. Under a blanket stability assumption, we provide a complete analysis to this problem. In particular, we establish uniqueness of the value function and verification result for optimal stationary Markov controls, in addition to the existence results. We also revisit this problem under a near-monotonicity condition but without any stability hypothesis. Our results also include policy improvement algorithms both in discrete and continuous time frameworks.

Risk-Sensitive Rear-Wheel Steering Control Method Based on the Risk Potential Field

Various driving assistance systems have been developed to reduce the number of automobile accidents. However, the control laws of these assistance systems differ based on each situation, and the discontinuous control command value may be input instantaneously. Therefore, a seamless and unified control law for driving assistance systems that can be used in multiple situations is necessary to realize more versatile autonomous driving. Although studies have been conducted on four-wheel steering that steers the rear wheels, these studies considered the role of the rear wheels only to improve vehicle dynamics and not to contribute to autonomous driving. Therefore, in this study, we define the risk potential field as a uniform control law and propose a rear-wheel steering control system that actively steers the rear wheels to contribute to autonomous driving, depending on the level of the perceived risk in the driving situation. The effectiveness of the proposed method is verified by a double lane change test, which is performed assuming emergency avoidance in simulations, and subject experiments using a driving simulator. The results indicate that actively steering the rear wheels ensures a safer and smoother drive while simultaneously improving the emergency avoidance performance.

Bayesian optimization assisted meal bolus decision based on Gaussian processes learning and risk-sensitive control

Effective postprandial glucose control is important to glucose management for subjects with diabetes mellitus. In this work, a data-driven meal bolus decision method is proposed without the need of subject-specific glucose management parameters. The postprandial glucose dynamics is learnt using Gaussian process regression. Considering the asymmetric risks of hyper- and hypoglycemia and the uncertainties in the predicted glucose trajectories, an asymmetric risk-sensitive cost function is designed. Bayesian optimization is utilized to solve the optimization problem, since the gradient of the cost function is unavailable. The proposed approach is evaluated using the 10-adult cohort of the FDA-accepted UVA/Padova T1DM simulator and compared with the standard insulin bolus calculator. For the case of announced meals, the proposed method achieves satisfactory and similar performance in terms of mean glucose and percentage time in [70, 180] mg/dL without increasing the risk of hypoglycemia. Similar results are observed for the case without the meal information (assuming that the patient follows a consistent diet) and the case of basal rate mismatches. In addition, a comparison with a run-to-run based method for the scenario of potentially incorrect meal carbohydrate counts is also performed, and the results show that the proposed method is more robust to the carbohydrate counting disturbances. At last, advisory-mode analysis is performed based on clinical data, which indicates that the method can determine safe and reasonable meal boluses in real clinical settings. The results verify the effectiveness and robustness of the proposed method and indicate the feasibility of achieving improved postprandial glucose regulation through a data-driven optimal control method.

Impedance Optimization for Uncertain Contact Interactions Through Risk Sensitive Optimal Control

This letter addresses the problem of computing optimal impedance schedules for legged locomotion tasks involving complex contact interactions. We formulate the problem of impedance regulation as a trade-off between disturbance rejection and measurement uncertainty. We extend a stochastic optimal control algorithm known as Risk Sensitive Control to take into account measurement uncertainty and propose a formal way to include such uncertainty for unknown contact locations. The approach can efficiently generate optimal state and control trajectories along with local feedback control gains, i.e. impedance schedules. Extensive simulations demonstrate the capabilities of the approach in generating meaningful stiffness and damping modulation patterns before and after contact interaction. For example, contact forces are reduced during early contacts, damping increases to anticipate a high impact event and tracking is automatically traded-off for increased stability. In particular, we show a significant improvement in performance during jumping and trotting tasks with a simulated quadruped robot.

In Between the $$LQG/H_{2}$$- and $$H_{\infty } $$-Control Theories

In this survey, we discuss various approaches to control theory that have arisen in the recent decades and reflect the desire to reach a trade-off between the $$LQG /H_2 $$ -control theory and the $$H_{\infty } $$ -control theory. The theories of the kind include the theory of risk-sensitive controllers, the theory of suboptimal control with a constraint on the $$H_{\infty } $$ -entropy functional, the mixed $$H_2/H_{\infty } $$ -control theory, the minimax $$LQG $$ -control theory, the anisotropy-based theory, and some others. The survey discusses in more detail the anisotropy-based control theory, which includes both the $$ LQG/H_2$$ - and the $$H_{\infty } $$ -theory within a single statement of the problem.