Feedback Policy Research Articles

Continuous-Time Mean Field Markov Decision Models

We consider a finite number of N statistically equal agents, each moving on a finite set of states according to a continuous-time Markov Decision Process (MDP). Transition intensities of the agents and generated rewards depend not only on the state and action of the agent itself, but also on the states of the other agents as well as the chosen action. Interactions like this are typical for a wide range of models in e.g. biology, epidemics, finance, social science and queueing systems among others. The aim is to maximize the expected discounted reward of the system, i.e. the agents have to cooperate as a team. Computationally this is a difficult task when N is large. Thus, we consider the limit for N→∞.\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$N\\rightarrow \\infty .$$\\end{document} In contrast to other papers we treat this problem from an MDP perspective. This has the advantage that we need less regularity assumptions in order to construct asymptotically optimal strategies than using viscosity solutions of HJB equations. The convergence rate is 1/N\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$1/\\sqrt{N}$$\\end{document}. We show how to apply our results using two examples: a machine replacement problem and a problem from epidemics. We also show that optimal feedback policies from the limiting problem are not necessarily asymptotically optimal.

On the Optimization Landscape of Dynamic Output Feedback Linear Quadratic Control

The convergence of policy gradient algorithms hinges on the optimization landscape of the underlying optimal control problem. Theoretical insights into these algorithms can often be acquired from analyzing those of linear quadratic control. However, most of the existing literature only considers the optimization landscape for static full-state or output feedback policies (controllers). We investigate the more challenging case of dynamic output-feedback policies for linear quadratic regulation (abbreviated as dLQR), which is prevalent in practice but has a rather complicated optimization landscape. We first show how the dLQR cost varies with the coordinate transformation of the dynamic controller and then derive the optimal transformation for a given observable stabilizing controller. One of our core results is the uniqueness of the stationary point of dLQR when it is observable, which provides an optimality certificate for solving dynamic controllers using policy gradient methods. Moreover, we establish conditions under which dLQR and linear quadratic Gaussian control are equivalent, thus providing a unified viewpoint of optimal control of both deterministic and stochastic linear systems. These results further shed light on designing policy gradient algorithms for more general decision-making problems with partially observed information.

Controlled descent training

SummaryIn this work, a novel and model‐based artificial neural network (ANN) training method is developed supported by optimal control theory. The method augments training labels in order to robustly guarantee training loss convergence and improve training convergence rate. Dynamic label augmentation is proposed within the framework of gradient descent training where the convergence of training loss is controlled. First, we capture the training behavior with the help of empirical Neural Tangent Kernels (NTK) and borrow tools from systems and control theory to analyze both the local and global training dynamics (e.g., stability, reachability). Second, we propose to dynamically alter the gradient descent training mechanism via fictitious labels as control inputs and an optimal state feedback policy. In this way, we enforce locally optimal and convergent training behavior. The novel algorithm, Controlled Descent Training (CDT), guarantees local convergence. CDT unleashes new potentials in the analysis, interpretation, and design of ANN architectures. The applicability of the method is demonstrated on standard regression and classification problems.

Closed-loop optimisation of neural networks for the design of feedback policies under uncertainty

Solving model predictive control (MPC) problems online can be computationally intractable, especially when considering uncertainty and nonlinear systems. One approach to avoid this is to train a neural network on a data-set of solutions of MPC problems (potentially nonlinear) offline, and to evaluate the trained control policy online. However, due to the separation of these optimisation problems the neural network controller must be carefully verified to ensure adequate closed loop performance. In this paper we propose a method to train a neural network in closed loop for control systems, in continuous or discrete time, while allowing for flexible consideration of parametric uncertainty. This method does not require off-line solutions of the NMPC problem, and instead directly optimises the desired closed loop performance. We prove that our method can approximate the optimal closed-loop control policy to arbitrary tolerance and in numerical examples demonstrate its performance compared to explicit MPC, imitation learning and nominal MPC.

Evaluation of prescription review and feedback policy on rational antibiotic use in primary healthcare settings in Beijing, China: a qualitative study using the Theoretical Domains Framework and the behaviour change wheel.

To decelerate antibiotic resistance driven by inappropriate antibiotic prescribing, a prescription review and feedback (PRF) policy is implemented in primary healthcare institutions (PHIs) in Beijing, China. However, evaluation of PRF implementation in PHIs is scarce. This study aims to systematically identify the barriers and facilitators of PRF policy implementation to provide evidence for antimicrobial stewardship. We conducted key informant interviews with 40 stakeholders engaged in the implementation of PRF in Beijing, including physicians, pharmacists and administrators. Interviews were audio recorded and transcribed verbatim. We coded the interview transcripts and mapped informant views to domains of the Theoretical Domains Framework. We then used a behaviour change wheel to suggest possible behavioural interventions. Procedural knowledge (Knowledge) and skills (Skill) of PRF were possessed by stakeholders. They felt responsible to promote the appropriate use of antibiotics (Social/professional role and identity) and believed that PRF could help to change inappropriate provider behaviours (Behavioural regulation) in prescribing antibiotics (Beliefs about consequences) under increased intention on antibiotic use (Stages of change). Moreover, informants called for a more unified review standard to enhance PRF implementation (Goals). Frequently identified barriers to PRF included inadequate capacity (Skill), using punishment mechanism (Behaviour regulation), reaching consistently lower antibiotic prescription rates (Goals), lack of resources (Environmental context and resources) and perceived pressure coming from patients (Social influences). Stakeholders believed that PRF implementation promoted the rational use of antibiotics at PHIs in Beijing. Still, PRF was hampered by inconsistencies in review process and resources needed for PRF implementation.

Reinforcement learning with model-based feedforward inputs for robotic table tennis

We rethink the traditional reinforcement learning approach, which is based on optimizing over feedback policies, and propose a new framework that optimizes over feedforward inputs instead. This not only mitigates the risk of destabilizing the system during training but also reduces the bulk of the learning to a supervised learning task. As a result, efficient and well-understood supervised learning techniques can be applied and are tuned using a validation data set. The labels are generated with a variant of iterative learning control, which also includes prior knowledge about the underlying dynamics. Our framework is applied for intercepting and returning ping-pong balls that are played to a four-degrees-of-freedom robotic arm in real-world experiments. The robot arm is driven by pneumatic artificial muscles, which makes the control and learning tasks challenging. We highlight the potential of our framework by comparing it to a reinforcement learning approach that optimizes over feedback policies. We find that our framework achieves a higher success rate for the returns (100% vs. 96%, on 107 consecutive trials, see https://youtu.be/kR9jowEH7PY) while requiring only about one tenth of the samples during training. We also find that our approach is able to deal with a variant of different incoming trajectories.

Risk-averse receding horizon motion planning for obstacle avoidance using coherent risk measures

This paper studies the problem of risk-averse receding horizon motion planning for agents with uncertain dynamics, in the presence of stochastic, dynamic obstacles. We propose a model predictive control (MPC) scheme that formulates the obstacle avoidance constraint using coherent risk measures. To handle disturbances, or process noise, in the state dynamics, the state constraints are tightened in a risk-aware manner to provide a disturbance feedback policy. We also propose a waypoint following algorithm that uses the proposed MPC scheme for discrete distributions and prove its risk-sensitive recursive feasibility while guaranteeing finite-time task completion. We further investigate some commonly used coherent risk metrics, namely, conditional value-at-risk (CVaR), entropic value-at-risk (EVaR), and g-entropic risk measures, and propose a tractable incorporation within MPC. We illustrate our framework via simulation studies.

Reinforcement Learning-Based Model-Free Controller for Feedback Stabilization of Robotic Systems.

This article presents a reinforcement learning (RL) algorithm for achieving model-free control of robotic applications. The RL functions are adapted with the least-square temporal difference (LSTD) learning algorithms to develop a model-free state feedback controller by establishing linear quadratic regulator (LQR) as a baseline controller. The classical least-square policy iteration technique is adapted to establish the boundary conditions for complexities incurred by the learning algorithm. Furthermore, the use of exact and approximate policy iterations estimates the parameters of the learning functions for a feedback policy. To assess the operation of the proposed controller, the trajectory tracking and balancing control problems of unmanned helicopters and balancer robotic applications are solved for real-time experiment. The results showed the robustness of the proposed approach in achieving trajectory tracking and balancing control.

On Optimal Coordinated Dispatch for Heterogeneous Storage Fleets With Partial Availability

This paper addresses the problem of optimal scheduling of an aggregated power profile (during a coordinated discharging or charging operation) by means of a heterogeneous fleet of storage devices subject to availability constraints. Devices have heterogeneous initial levels of energy, power ratings and efficiency; moreover, the fleet operates without cross-charging of the units. An explicit feedback policy is proposed to compute a feasible schedule whenever one exists and scalable design procedures to achieve maximum time to failure or minimal unserved energy in the case of unfeasible aggregated demand profiles. Finally, a time-domain characterization of the set of feasible demand profiles using aggregate constraints is proposed, suitable for optimization problems where the aggregate population behaviour is of interest.

Data-Driven Decentralized Control for Large-Scale Systems With Sparsity and Communication Delays

This article studies the decentralized control of large-scale systems with sparsity and communication delays. The large-scale system is defined over a directed connected graph and the information structure is partially nested. Based on the decomposition of the noise history, the optimal problem of the overall large-scale system can be decomposed into independent subproblems. Hence, the data-driven decentralized control method is investigated to find the optimal controllers using adaptive dynamic programming (ADP), which could release the dependence on the knowledge of model. In addition, state feedback and output feedback policy iteration algorithms are developed, respectively. Rigorous stability analysis shows that the proposed algorithms can stabilize the large-scale systems asymptotically. Finally, the effectiveness of the proposed theoretical methods is demonstrated by the application of heavy duty vehicle (HDV) platooning.

Educators’ pre-Covid19 lived experience of the assessment and feedback policy at a UK higher education institution

This study was conducted before the Covid19 pandemic. It used an interpretative thematic analysis approach to inquire about educators’ experiences of assessment and feedback policy and their views on assessment feedback. Eighteen teaching staff from a UK-based higher education institution expressed their opinions on various factors that facilitated or hindered the adoption of the assessment and feedback policy, along with their perceptions of assessment and feedback. The policy required that teaching staff provide feedback using technology such as Turnitin, with a two-week turnaround. Three themes were developed from the interpretative thematic analysis: 1) assessment and feedback policy as a personal and professional challenge, 2) mixed perceptions on the effectiveness of feedback, and 3) facilitating conditions for a successful assessment and feedback policy implementation. Workload and time constraints appeared to be the most prominent issues across all the interviews, affecting educators at every level of their professional and personal experiences. Insufficient time for attending meetings, providing timely and quality feedback, conducting research, pursuing professional development, and engaging in innovative strategies created tension between educators’ engagement and commitment to enhancing students learning and the pressure imposed by assessment and feedback policy demands. This research has implications for both educators and their professional development. Additionally, it can inform the university’s internal development of assessment practices and possibly broaden the understanding of assessment and feedback, particularly regarding the potential mismatch between the assessment and feedback policy and educators’ experiences and expectations.

Explicit model predictive control through robust optimization

AbstractA strategy that calculates an explicit state feedback policy to regulate constrained uncertain discrete‐time uncertain linear systems is presented. We consider uncertain processes, affected by box‐bounded multiplicative uncertainty as well as bounded additive uncertainty with linear state and inputs constraints. The proposed method includes (i) the calculation of a terminal set constraint and (ii) the robust reformulation of state constraints in the prediction horizon. These features allow the derivation of the desired policy by solving a single multiparametric quadratic programming problem that guarantees feasible operation in the presence of uncertainty. Additionally, we employ variable and constraint elimination approaches to enhance the computational performance of the strategy. We demonstrate the steps and benefits of these developments with a numerical example and a chemical engineering case study.

Maximum-Entropy Multi-Agent Dynamic Games: Forward and Inverse Solutions

In this article, we study the problem of multiple stochastic agents interacting in a dynamic game scenario with continuous state and action spaces. We define a new notion of stochastic Nash equilibrium for boundedly rational agents, which we call the entropic cost equilibrium (ECE). We show that ECE is a natural extension to multiple agents of maximum entropy optimality for a single agent. We solve both the “forward” and “inverse” problems for the multi-agent ECE game. For the forward problem, we provide a Riccati algorithm to compute closed-form ECE feedback policies for the agents, which are exact in the linear-quadratic-gaussian case. We give an iterative variant to find locally ECE feedback policies for the nonlinear case. For the inverse problem, we present an algorithm to infer the cost functions of the multiple interacting agents given noisy, boundedly rational input and state trajectory examples from agents acting in an ECE. The effectiveness of our algorithms is demonstrated in a simulated multi-agent collision avoidance scenario, and with data from the INTERACTION traffic dataset. In both cases, we show that, by taking into account the agents' game theoretic interactions using our algorithm, a more accurate model of agents' costs can be learned, compared with standard inverse optimal control methods.

Efficient Anytime CLF Reactive Planning System for a Bipedal Robot on Undulating Terrain

We propose and experimentally demonstrate a reactive planning system for bipedal robots on unexplored, challenging terrain. The system includes: a multilayer local map for assessing traversability; an anytime omnidirectional control Lyapunov function for use with a rapidly exploring random tree star (RRT*) that generates a vector field for specifying motion between nodes; a subgoal finder when the final goal is outside of the current map; and a finite-state machine to handle high-level mission decisions. The system also includes a reactive thread that copes with robot deviations via a vector field, defined by a closed-loop feedback policy. The vector field provides real-time control commands to the robot's gait controller as a function of instantaneous robot pose. The system is evaluated on various challenging outdoor terrains and cluttered indoor scenes in both simulation and experiment on Cassie Blue, a bipedal robot with 20 degrees of freedom. All implementations are coded in C++ with the robot operating system and are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/UMich-BipedLab/CLF_reactive_planning_system</uri> .

Impact of Effective Performance Appraisal Systems on Employee Motivation and Retention in Private Banking Industry of Bangladesh

Aims: Healthy economic development requires properly managing the banking industry of any country. Along with state-owned banks, private banks play a critical role in the country's economy. Managers in all types of banks now confront the same challenge: how to get the utmost output from their employees. Therefore, Performance appraisal appears to be inevitable since it set the standard for comparing actual performance to established objectives and recommending practical solutions that help the organization achieve sustainable growth. Therefore, the purpose of this research is to determine the effect of performance appraisal on employee motivation and retention.&#x0D; Place and Duration of Study: The study was conducted in the private banking industry of Bangladesh between November 01, 2022, to March 30, 2023. The study duration was determined based on the scope of the research project and the availability of time and resources to complete it within the designated timeframe.&#x0D; Methodology: A random sampling method was used to conduct the study through a structured questionnaire among 110 employees from various private banks. Questionnaires were used to collect quantitative data, which was then analyzed using the Statistical Package for Social Sciences (SPSS) 25.0 software.&#x0D; Results: Major findings of the study indicate that performance appraisal is critical for employee motivation and employees perceived performance appraisal as a fair procedure, which increased their motivation and retention. Compensation and reward, and feedback are also perceived positively by the employees, which motivate them to stay in the banks.&#x0D; Conclusion: There is a recommendation that banks should review the appraisal process regularly, improve the feedback policy, introduce an effective reward system, provide the opportunity to employees for career development and empowerment, and maintain a participative environment.

Risk-averse dynamic pricing using mean-semivariance optimization

In many revenue management applications risk-averse decision-making is crucial. In dynamic settings, however, it is challenging to find the right balance between maximizing expected rewards and avoiding poor performances. In this paper, we consider time-consistent mean-semivariance (MSV) optimization for dynamic pricing problems within a discrete MDP framework, which are shown to be NP hard. We present a novel fixpoint-based dynamic programming approach to compute risk-sensitive feedback policies with Pareto-optimal combinations of mean and semivariance. We illustrate the effectiveness and the applicability of our concepts compared to state-of-the-art heuristics. For various numerical examples the results show that our approach clearly outperforms all other heuristics and obtains a performance guarantee with less then 0.2% optimality gap. Our approach is general and can be applied to MDPs beyond dynamic pricing.

Investment subsidies and redistributive capital income taxation in a neoclassical growth model

AbstractHow do investment subsidies bear on pure redistribution when coupled with capital income taxes? In a heterogeneous agent, neoclassical growth framework it is found that on impact, with no optimizing behavior, investment subsidies are good for growth but bad for redistribution. The opposite holds for capital income taxes. But when the government acts as a Stackelberg leader vis‐à‐vis the private sector (the follower), the optimal feedback policy is by construction time‐consistent and implies that in a long‐run optimum the tax scheme does not distort accumulation. This holds regardless of social preferences. For the feedback Stackelberg equilibrium I find that (pure) redistribution can go either way and capital income taxes are nonzero in the long‐run, time‐consistent optimum, depending on the social weight of those who receive redistributive transfers, the distribution of pretax factor incomes, and the intertemporal elasticity of substitution. It is argued that investment subsidies may be an important indirect tool for redistribution, and may allow for the separation of “efficiency” and “equity” concerns.

Implicit feedback policies for COVID-19: why “zero-COVID” policies remain elusive

Successful epidemic modeling requires understanding the implicit feedback control strategies used by populations to modulate the spread of contagion. While such strategies can be replicated with intricate modeling assumptions, here we propose a simple model where infection dynamics are described by a three parameter feedback policy. Rather than model individuals as directly controlling the contact rate which governs the spread of disease, we model them as controlling the contact rate’s derivative, resulting in a dynamic rather than kinematic model. The feedback policy used by populations across the United States which best fits observations is proportional-derivative control, where learned parameters strongly correlate with observed interventions (e.g., vaccination rates and mobility restrictions). However, this results in a non-zero “steady-state” of case counts, implying current mitigation strategies cannot eradicate COVID-19. Hence, we suggest making implicit policies a function of cumulative cases, resulting in proportional-integral-derivative control with higher potential to eliminate COVID-19.

In search of learning-focused feedback practices: a linguistic analysis of higher education feedback policy

Feedback can have one of the biggest positive influences on higher education learners. Despite this, teachers and students consistently report being dissatisfied with feedback. In response, there has been a theoretical shift in how feedback is conceptualised and discussed within the research literature. Older transmission-focused models have evolved into more learning-focused approaches. However, the extent to which higher education feedback policy, and subsequent practice, embrace such current thinking is unclear. This research adopted a corpus linguistics approach to analyse how the term ‘feedback’ was used within 50 UK higher education institutions’ feedback policy texts. Sketch Engine was used to analyse ‘feedback’ collocation frequencies. To investigate differences between research-intensive (Russell Group) and more teaching-focused (non-Russell Group) universities, separate corpora were also compiled and compared. Quantitative results showed that the most frequent feedback collocations related to outdated transmission-focused feedback practices. However, qualitative deductive thematic analysis found that many feedback policies did present learning-focused feedback practices despite using transmission-focused language. Feedback appears to mean different things to different higher education institutions which could lead to confusion for teachers and students. The research concludes by presenting key practical implications for practitioners involved in feedback policy design and enactment to improve practice.

Output feedback Q-learning for discrete-time finite-horizon zero-sum games with application to the [formula omitted] control

In this paper, we present a Q-learning framework for solving finite-horizon zero-sum game problems involving the H∞ control of linear system without knowing the dynamics. Research in the past mainly focused on solving problems in infinite horizon with completely measurable state. However, in the practical engineering, the system state is not always directly accessible, and it is difficult to solve the time-varying Riccati equation associated with the finite-horizon setting directly either. The main contribution of the proposed model-free algorithm is to determine the optimal output feedback policies without measurement state in finite-horizon setting. To achieve this goal, we first describe the Q-function caused by finite-horizon problems in the context of state feedback, then we parameterize the Q-functions as input–output vectors functions. Finally, the numerical examples on aircraft dynamics demonstrate the algorithm’s efficiency.