Closed-loop Policy Research Articles

Can control aid in attaining sustainable goals? An improved data-informed framework to promote shared mobility

This work explores the role control theory and practices can play in nudging a virtuous shift in mobility habits (e.g. from vehicle ownership to usership), while keeping individual inclinations and needs in the (control) loop using data. To this end, an EU-wide survey is leveraged with a two-fold objective. First, a fine-grained characterization of individuals (the Sharing-DNA) is provided, dictated by the main socio-economic drivers for adopting shared mobility solutions extracted from the data. Second, a network is constructed to mimic mutual influences among individuals based on information on mobility patterns extrapolated from the EU survey. These ingredients are merged into an irreversible cascade model, representing the stepping stone for designing optimal, human-centered incentive policies to nudge the use of shared mobility services. In the quest to attain a trade-off between usage maximization and investment containment, the tuning of the design knobs is explored accounting for individuals’ centrality within the influence network. An extensive set of simulated scenarios demonstrate the potential benefits of individualized closed-loop policies in promoting sustainable mobility choices while respecting individualities, showcasing how the proposed framework can become an actionable tool for policymakers toward a more sustainable future.

Hierarchical Deep Reinforcement Learning for Self-Powered Monitoring and Communication Integrated System in High-Speed Railway Networks

To align with the vision of future intelligent high-speed railway (HSR) networks, integrating sensor monitoring and remote communication are challenging for ensuring the lightweight of train equipment, high-quality transmissions, and dynamic interaction between monitoring and communication. In this paper, we propose a self-powered multisensor monitoring and communication integrated system in HSR. A low-power backscatter communication working framework of the self-powered monitoring system is designed in the monitoring network model, and a finite Gaussian mixture model (GMM) clustering method is used to analyze the communication cell coverage area in the communication network model. Aiming to minimize the total task completion time, we formulate a data monitoring and remote communication problem with the energy transfer constraint, data collection constraint, and transmission data rate constraint. As for the non-convex minimum time optimization problem, we develop a novel option-based hierarchical deep reinforcement learning (OHDRL) method to deal with the complex continuous variation characteristics of the monitoring and communication integrated HSR system. The system learns to select options at a high level, and the action is executed according to the policy of the selected option at a low level. This approach enables us to handle stochastic HSR environments, closed-loop policies, and goals in a temporal abstraction way. Numerical results reveal that the proposed algorithm for the integrated monitoring and communication HSR achieves a significantly higher reward and more stable learning performance than other algorithms in the literature.

How Bad is Selfish Driving? Bounding the Inefficiency of Equilibria in Urban Driving Games

We consider the interaction among agents engaging in a driving task and we model it as general-sum game. This class of games exhibits a plurality of different equilibria posing the issue of equilibrium selection. While selecting the most efficient equilibrium (in term of social cost) is often impractical from a computational standpoint, in this work we study the (in)efficiency of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">any</i> equilibrium players might agree to play. More specifically, we bound the equilibrium inefficiency by modeling driving games as particular type of congestion games over spatio-temporal resources. We obtain novel guarantees that refine existing bounds on the Price of Anarchy (PoA) as a function of problem-dependent game parameters. For instance, the relative trade-off between proximity costs and personal objectives such as comfort and progress. Although the obtained guarantees concern open-loop trajectories, we observe efficient equilibria even when agents employ closed-loop policies trained via decentralized multi-agent reinforcement learning.

Fostering the use of sharing mobility solutions via control-oriented policy design

In the quest for reducing greenhouse gas (GHG) emissions, mobility plays a lion's role. In particular, moving from ownership to usership has profound individual implications, as vehicles are in many cultures proxy of social status and power symbols. To sustain the shift to shared mobility, we use data to extrapolate the main socio-economic drivers that guide the adoption of this model, the so-called Sharing-DNA, ultimately building a dynamical model characterizing the evolution of individual inclinations over time. This novel representation allows us to exploit optimal control tools for the design of innovative human-centric policies to foster the adoption of sharing mobility solutions. The results here presented demonstrate the potential impact of individualized closed-loop policies in promoting this crucial behavioral change.

Learning Closed-Loop Dough Manipulation Using a Differentiable Reset Module

Deformable object manipulation has many applications such as cooking and laundry folding in our daily lives. Manipulating elastoplastic objects such as dough is particularly challenging because dough lacks a compact state representation and requires contact-rich interactions. We consider the task of flattening a piece of dough into a specific shape from RGB-D images. While the task is seemingly intuitive for humans, there exist local optima for common approaches such as naive trajectory optimization. We propose a novel trajectory optimizer that optimizes through a differentiable “reset” module, transforming a single-stage, fixed-initialization trajectory into a multistage, multi-initialization trajectory where all stages are optimized jointly. We then train a closed-loop policy on the demonstrations generated by our trajectory optimizer. Our policy receives partial point clouds as input, allowing ease of transfer from simulation to the real world. We show that our policy can perform real-world dough manipulation, flattening a ball of dough into a target shape.

Primal–dual differential dynamic programming: A model-based reinforcement learning for constrained dynamic optimization

The main objective of this study is to develop primal–dual differential dynamic programming (DDP), a model-based reinforcement learning (RL) framework that can handle constrained dynamic optimization problems. DDP has advantages of being able to provide a closed-loop policy and having computational complexity that grows linearly with respect to the time horizon. To take advantage, the DDP should consider optimality and feasibility for the disturbed state during closed-loop operations. Previous DDPs consider the feasibility only for the nominal state condition and can handle limited types of constraints. In this paper, we propose a primal–dual DDP incorporating modified augmented Lagrangian that can handle general nonlinear constraints. We pay special attention to obtain the feasible policy when active set changes due to the state perturbations, using path-following predictor–corrector approach. The developed framework method was applied to van der Pol oscillator and batch crystallization process, thereby validating the key aspects of this study.

EPSILON: An Efficient Planning System for Automated Vehicles in Highly Interactive Environments

In this article, we present an efficient planning system for automated vehicles in highly interactive environments (EPSILON). EPSILON is an efficient interaction-aware planning system for automated driving, and is extensively validated in both simulation and real-world dense city traffic. It follows a hierarchical structure with an interactive behavior planning layer and an optimization-based motion planning layer. The behavior planning is formulated from a partially observable Markov decision process (POMDP), but is much more efficient than naively applying a POMDP to the decision-making problem. The key to efficiency is guided branching in both the action space and observation space, which decomposes the original problem into a limited number of closed-loop policy evaluations. Moreover, we introduce a new driver model with a safety mechanism to overcome the risk induced by the potential imperfectness of prior knowledge. For motion planning, we employ a spatio-temporal semantic corridor (SSC) to model the constraints posed by complex driving environments in a unified way. Based on the SSC, a safe and smooth trajectory is optimized, complying with the decision provided by the behavior planner. We validate our planning system in both simulations and real-world dense traffic, and the experimental results show that our EPSILON achieves human-like driving behaviors in highly interactive traffic flow smoothly and safely without being overconservative compared to the existing planning methods.

Throughput Optimal Routing in Blockchain-Based Payment Systems

Cryptocurrency networks, such as Bitcoin, have emerged as a distributed alternative to traditional centralized financial transaction networks. However, there are major challenges in scaling up the throughput of such networks. Lightning network and Spider network are alternates that build bidirectional payment channels on top of cryptocurrency networks using smart contracts, to enable fast transactions that bypass the Blockchain. In this article, we study the problem of routing transactions in such a payment processing network. We first propose a Stochastic model to study such a system, as opposed to a fluid model that is studied in the literature. Each link in such a model is a two-sided queue, and unlike classical queues, such queues are not stable unless there is an external control. We propose a notion of stability for the payment processing network consisting of such two-sided queues using the notion of on-chain rebalancing. We then characterize the capacity region and propose a throughput optimal algorithm that stabilizes the system under any load within the capacity region. The stochastic model enables us to study closed-loop policies, which typically have better queuing/delay performance than the open-loop policies (or static split rules) studied in the literature. We investigate this through simulations.

Safety and Control Analysis of Hybrid Liquid-liquid Extraction and Divided Wall Column for Biobutanol Purification

Butanol is getting attention among the possible biofuels as an alternative to bioethanol. Compared to bioethanol, biobutanol shows less miscibility with water, less flammability and corrosivity, and it has the advantage of being able to directly replace gasoline in the car engines without any modifications. However, the production cost from the Acetone-Butanol-Ethanol (ABE) fermentation process remains high, mainly due to the low yield of butanol. The conventional recovery of butanol by distillation is a process of high energy consumption that has greatly restricted its industrial production. In this work different hybrid configurations, based on liquid-liquid extraction and dividing wall columns were examined. Sustainability was analysed based on green metrics, inherent security, and control properties through open and closed-loop policy using PI controllers. The results indicate that as long as the process is highly intensified, the sustainability and the inherent safety are improved but not necessarily the control properties. The best alternative was selected as a compromise between contrasting objective functions.

Research on the Supervision of vehicle 4G Video Surveillance platform in Freight Company

In order to solve the difficult problem of dynamic driving detection in vehicle transportation for a long time, Jiaozuo Transportation Bureau took the lead in using 4G platform for dynamic supervision of driver behavior, so as to reduce driving accidents. According to the data provided by Jiaozuo freight company, this paper analyzes the internal relationship among vehicle qualified rate, 4G video vehicle monitoring and evaluation project score and alarm record. And according to the relationship between the three, we use SPSS software to analyze the correlation and cluster of freight companies, and study the closed-loop disposal and company policy of companies with good operation.

How good are default investment policies in defined contribution pension plans?

AbstractDefined contribution (DC) pension plans have been gaining ground in the last 10–20 years as the preferred system for many countries and other agencies, both private and public. The central question for a DC plan is how to invest in order to reach the participant's retirement goals. Given the financial illiteracy of the general population, it is common to offer a default policy for members who do not actively make investment choices. Using data from the Chilean system, we discuss an investment model with fixed contribution rates and compare the results with the existing default policy under multiple objectives. Our results indicate that the Chilean default policy has good overall performance, but specific closed-loop policies have a higher probability of achieving desired retirement goals and can reduce the expected shortfall at retirement.

Data-driven feedback algorithms for automated position identification and environment reconstruction of autonomous vehicles

Abstract Independently of the automation level challenging aspects on unmanned vehicle management such as performance optimization processes, adaptive cruise control systems, steering and braking controls, are related to the assessment of the operational functioning degree of relevant responsive procedures. This paper states the problem of determining both the position and the environment of fully autonomous cars for safe and efficient car routing control. An adaptive, closed loop policy is proposed utilizing a minimum number of measurements available by principal car technology, a laser sensor and a smart camera. No additional investment is required on other devices where the costly continuous real time scans can be now reduced. The low computational complexity encourages for embedded designs providing real time responses. Missing or invalid sensor information is estimated by a first approach using neural network programming. Positions of mobile and immobile entities are determined and the related environmental context within the desired vehicle vicinity is precisely reproduced. The performance of the proposed methodology is evaluated through realistic simulation data fed by multiple ground truth recordings. Hence, multiple biases on direct information as generated by professional drivers leading to untruthful and unreliable conclusions are now minimized. A trajectory within Paris central areas and suburbs is considered involving varying traffic conditions and rich road infrastructure. Some first qualitative and quantitative results appraise the accuracy and performance of the proposed methodology while further research targets are discussed.

How Good Are Default Investment Policies in Defined Contribution Pension Plans?

Defined contribution (DC) pension plans have been gaining ground in the last 10-20 years as the preferred system for many countries and other agencies, both private and public. The central question for a DC plan is how to invest in order to reach the participant's retirement goals. Given the financial illiteracy of the general population, it is common to offer a default policy for members who do not actively make investment choices. Using data from the Chilean system, we discuss an investment model with fixed contribution rates and compare the results with the default policy under multiple objectives. Our results indicate that the Chilean default policy has a good overall performance, but specific closed-loop policies have a higher probability of achieving desired retirement goals and can reduce the expected shortfall at retirement.

A performance bound for optimal insulin infusion in individuals with Type 1 diabetes ingesting a meal with slow postprandial response

Recent work, Goodwin et al. (2015, 2018), has established that, subject to an assumption regarding the relative blood glucose response times of meal and insulin, it is optimal for the minimisation of the peak blood glucose excursion to apply a bolus of insulin at the time of (or slightly prior to) meal ingestion. A key assumption underlying this result is that the impact of an impulse of insulin “lasts longer” than the impact of an impulse of food. The result is consistent with clinical trials reported in Lopez et al. (2014). In the current paper we consider the converse situation where the response of the meal lasts longer than the response of an insulin impulse. The latter situation is known to arise in certain circumstances e.g., when low glycaemic index and/or hi-fat/hi-protein meals are consumed (Bell et al., 2015; Lodefalk et al., 2008; van der Hoogt et al., 2017). We consider the same set-up as in Goodwin et al. (2015). However, here we show that, when the response of the meal lasts longer than the response of an insulin impulse, it is optimal to use an open-loop policy combining an insulin bolus (applied with the meal) and a specific form of decaying insulin flow thereafter. This gives support to results for clinical trials, reported in Lopez et al. (2017). However, as far as we are aware, no earlier work has given a methodology to quantify the best insulin flow after the initial bolus nor the best split between bolus and post-bolus insulin flow. In support of the theory, we apply the results to a model obtained from clinical trials on a real subject with Type 1 diabetes and include a discussion of open-loop versus closed-loop policies.

Model-Based Reinforcement Learning for Closed-Loop Dynamic Control of Soft Robotic Manipulators

Dynamic control of soft robotic manipulators is an open problem yet to be well explored and analyzed. Most of the current applications of soft robotic manipulators utilize static or quasi-dynamic controllers based on kinematic models or linearity in the joint space. However, such approaches are not truly exploiting the rich dynamics of a soft-bodied system. In this paper, we present a model-based policy learning algorithm for closed-loop predictive control of a soft robotic manipulator. The forward dynamic model is represented using a recurrent neural network. The closed-loop policy is derived using trajectory optimization and supervised learning. The approach is verified first on a simulated piecewise constant strain model of a cable driven under-actuated soft manipulator. Furthermore, we experimentally demonstrate on a soft pneumatically actuated manipulator how closed-loop control policies can be derived that can accommodate variable frequency control and unmodeled external loads.

Comparing Task Simplifications to Learn Closed-Loop Object Picking Using Deep Reinforcement Learning

Enabling autonomous robots to interact in unstructured environments with dynamic objects requires manipulation capabilities that can deal with clutter, changes, and objects' variability. This paper presents a comparison of different reinforcement learning-based approaches for object picking with a robotic manipulator. We learn closed-loop policies mapping depth camera inputs to motion commands and compare different approaches to keep the problem tractable, including reward shaping, curriculum learning and using a policy pre-trained on a task with a reduced action set to warm-start the full problem. For efficient and more flexible data collection, we train in simulation and transfer the policies to a real robot. We show that using curriculum learning, policies learned with a sparse reward formulation can be trained at similar rates as with a shaped reward. These policies result in success rates comparable to the policy initialized on the simplified task. We could successfully transfer these policies to the real robot with only minor modifications of the depth image filtering. We found that using a heuristic to warm-start the training was useful to enforce desired behavior, while the policies trained from scratch using a curriculum learned better to cope with unseen scenarios where objects are removed.

Closed-Loop Policies for Operational Tests of Safety-Critical Systems

Manufacturers of safety-critical systems must make the case that their product is sufficiently safe for public deployment. Much of this case often relies upon critical event outcomes from real-world testing, requiring manufacturers to be strategic about how they allocate testing resources in order to maximize their chances of demonstrating system safety. This paper frames the partially observable and the belief-dependent problem of test scheduling as a Markov decision process, which can be solved efficiently to yield closed-loop manufacturer testing policies. By solving for policies over a wide range of problem formulations, we are able to provide high-level guidance for manufacturers and regulators on issues relating to the testing of safety-critical systems. This guidance spans an array of topics, including circumstances under which manufacturers should continue testing despite observed incidents, when manufacturers should test aggressively, and when regulators should increase or reduce the real-world testing requirements for safety-critical systems.

Stochastic differential games and inverse optimal control and stopper policies

ABSTRACTIn this paper, we consider a two-player stochastic differential game problem over an infinite time horizon where the players invoke controller and stopper strategies on a nonlinear stochastic differential game problem driven by Brownian motion. The optimal strategies for the two players are given explicitly by exploiting connections between stochastic Lyapunov stability theory and stochastic Hamilton–Jacobi–Isaacs theory. In particular, we show that asymptotic stability in probability of the differential game problem is guaranteed by means of a Lyapunov function which can clearly be seen to be the solution to the steady-state form of the stochastic Hamilton–Jacobi–Isaacs equation, and hence, guaranteeing both stochastic stability and optimality of the closed-loop control and stopper policies. In addition, we develop optimal feedback controller and stopper policies for affine nonlinear systems using an inverse optimality framework tailored to the stochastic differential game problem. These results are then used to provide extensions of the linear feedback controller and stopper policies obtained in the literature to nonlinear feedback controllers and stoppers that minimise and maximise general polynomial and multilinear performance criteria.

A comparison of open-loop and closed-loop strategies in colloidal self-assembly

Ordered colloidal crystals possess unique photonic properties for a wide range of applications in engineering, material science, communications, and medicine. However, ordered structures are hard to achieve even in small systems, due to the formation of defects during the process. In this paper, we focus on the control of a micron-sized SiO2 colloidal self-assembly process for defect-free crystal in an externally applied electric field batch system. Five control strategies including time-constant and time-varying, heuristic and model-based open-loop and closed-loop policies are investigated, to understand their control mechanisms. The model-based policies are designed using a reduced-order Markov state model, built with samples from Brownian dynamics simulations, and are calculated with dynamic programming. The performance of all the five strategies are evaluated on both the Brownian dynamics simulation and the experiments. Time-varying control strategies can improve the yield of grain-boundary-free crystals over that of their time-constant counterparts. Moreover, using feedback as “endpoint detection” to terminate the process can also shorten the process time, compared to open-loop strategies.

Collective Multiagent Sequential Decision Making Under Uncertainty

Multiagent sequential decision making has seen rapid progress with formal models such as decentralized MDPs and POMDPs. However, scalability to large multiagent systems and applicability to real world problems remain limited. To address these challenges, we study multiagent planning problems where the collective behavior of a population of agents affects the joint-reward and environment dynamics. Our work exploits recent advances in graphical models for modeling and inference with a population of individuals such as collective graphical models and the notion of finite partial exchangeability in lifted inference. We develop a collective decentralized MDP model where policies can be computed based on counts of agents in different states. As the policy search space over counts is combinatorial, we develop a sampling based framework that can compute open and closed loop policies. Comparisons with previous best approaches on synthetic instances and a real world taxi dataset modeling supply-demand matching show that our approach significantly outperforms them w.r.t. solution quality.