Hindsight Optimization Research Articles

Evaluating Human–Robot Interaction Algorithms in Shared Autonomy via Quality Diversity Scenario Generation

The growth of scale and complexity of interactions between humans and robots highlights the need for new computational methods to automatically evaluate novel algorithms and applications. Exploring diverse scenarios of humans and robots interacting in simulation can improve understanding of the robotic system and avoid potentially costly failures in real-world settings. We formulate this problem as a quality diversity (QD) problem, of which the goal is to discover diverse failure scenarios by simultaneously exploring both environments and human actions. We focus on the shared autonomy domain, in which the robot attempts to infer the goal of a human operator, and adopt the QD algorithms CMA-ME and MAP-Elites to generate scenarios for two published algorithms in this domain: shared autonomy via hindsight optimization and linear policy blending. Some of the generated scenarios confirm previous theoretical findings, while others are surprising and bring about a new understanding of state-of-the-art implementations. Our experiments show that the QD algorithms CMA-ME and MAP-Elites outperform Monte-Carlo simulation and optimization-based methods in effectively searching the scenario space, highlighting their promise for automatic evaluation of algorithms in human–robot interaction.

Cover’s Rebalancing Option with Discrete Hindsight Optimization

The author studies T. Cover’s rebalancing option (Ordentlich and Cover 1998) under discrete hindsight optimization in continuous time. The payoff in question is equal to the final wealth that would have accrued to an initial deposit of 1 unit of the numéraire into the best of some finite set of (perhaps levered) rebalancing rules determined in hindsight. A rebalancing rule (or fixed-fraction betting scheme) amounts to fixing an asset allocation (i.e., 200% equities and −100% bonds) and then continuously executing rebalancing trades so as to counteract allocation drift. Restricting the hindsight optimization to a small number of rebalancing rules (i.e., 2) has some advantages over the pioneering approach taken by Cover &amp; Company in their theory of <i>universal portfolios</i> (1986, 1991, 1996, 1998), wherein one’s trading performance is benchmarked relative to the final wealth of the best unlevered rebalancing rule (of any kind) in hindsight. Our approach lets practitioners express an <i>a priori</i> view that one of the favored asset allocations (“bets”) in the set {<i>b</i>₁, …, <i>b_n</i>} will turn out to have performed spectacularly well in hindsight. In limiting our robustness to some discrete set of asset allocations (rather than all possible asset allocations), we reduce the price of the rebalancing option and guarantee that we will achieve a correspondingly higher percentage of the hindsight-optimized wealth at the end of the planning period. A practitioner who lives to delta-hedge this variant of Cover’s rebalancing option through several decades is guaranteed to see the day that his realized compound-annual capital growth rate is very close to that of the best <i>b_i</i> in hindsight, hence the point of the rock-bottom option price.

Positional Flexibility and Consistent Assignment in Long-Term Care Rostering

We consider the rostering decisions of a long-term care facility that assigns a set of interchangeable caregivers working a particular shift to units, and evaluate whether part-time or full-time workers should have higher priority to work in their “home” units. The facility’s objective is to minimize the monthly inconsistency level – i.e., a widely promoted quality metric representing the number of different caregivers working in each unit. We introduce simple rostering heuristics that prioritize part-time or full-time workers, and present a stochastic model of the repeated rostering problem to analytically compare their performance. This analysis supports prioritizing the assignment of part-time workers to their home units, while flexibly assigning full-time workers to minimize the inconsistency level. Using data from over 15,000 shifts worked by certified nursing assistants at three nursing homes, we compare the actual schedules to the consistency-maximizing schedules using hindsight optimization and test the performance of our rostering heuristics via trace-based simulation. This reinforces the superiority of prioritizing part-time workers, observing reductions in the inconsistency level between 20% and 30% in trace-based simulations of historical schedules. We also establish an upper bound for a threshold on the time horizon above which a policy giving assignment priority to part-time workers outperforms one giving priority to full-time workers.

Cover's Rebalancing Option With Discrete Hindsight Optimization

We study T. Cover's rebalancing option (Ordentlich and Cover 1998) under discrete hindsight optimization in continuous time. The payoff in question is equal to the final wealth that would have accrued to a $\$1$ deposit into the best of some finite set of (perhaps levered) rebalancing rules determined in hindsight. A rebalancing rule (or fixed-fraction betting scheme) amounts to fixing an asset allocation (i.e. $200\%$ stocks and $-100\%$ bonds) and then continuously executing rebalancing trades to counteract allocation drift. Restricting the hindsight optimization to a small number of rebalancing rules (i.e. 2) has some advantages over the pioneering approach taken by Cover $\&$ Company in their brilliant theory of universal portfolios (1986, 1991, 1996, 1998), where one's on-line trading performance is benchmarked relative to the final wealth of the best unlevered rebalancing rule of any kind in hindsight. Our approach lets practitioners express an a priori view that one of the favored asset allocations (bets) $b\in\{b_1,...,b_n\}$ will turn out to have performed spectacularly well in hindsight. In limiting our robustness to some discrete set of asset allocations (rather than all possible asset allocations) we reduce the price of the rebalancing option and guarantee to achieve a correspondingly higher percentage of the hindsight-optimized wealth at the end of the planning period. A practitioner who lives to delta-hedge this variant of Cover's rebalancing option through several decades is guaranteed to see the day that his realized compound-annual capital growth rate is very close to that of the best $b_i$ in hindsight. Hence the point of the rock-bottom option price.

Shared autonomy via hindsight optimization for teleoperation and teaming

In shared autonomy, a user and autonomous system work together to achieve shared goals. To collaborate effectively, the autonomous system must know the user’s goal. As such, most prior works follow a predict-then-act model, first predicting the user’s goal with high confidence, then assisting given that goal. Unfortunately, confidently predicting the user’s goal may not be possible until they have nearly achieved it, causing predict-then-act methods to provide little assistance. However, the system can often provide useful assistance even when confidence for any single goal is low (e.g. move towards multiple goals). In this work, we formalize this insight by modeling shared autonomy as a partially observable Markov decision process (POMDP), providing assistance that minimizes the expected cost-to-go with an unknown goal. As solving this POMDP optimally is intractable, we use hindsight optimization to approximate. We apply our framework to both shared-control teleoperation and human–robot teaming. Compared with predict-then-act methods, our method achieves goals faster, requires less user input, decreases user idling time, and results in fewer user–robot collisions.

Exact Replication of the Best Rebalancing Rule in Hindsight

This paper prices the financial derivative whose payoff is the wealth that would have accrued to a $1 deposit into the best continuously-rebalanced portfolio in hindsight. For the market with a risk-free bond and a single stock in geometric Brownian motion, Cover and Ordentlich (1998) priced this derivative at time-0, for unlevered hindsight optimization. This paper reports the price under levered hindsight optimization, at any time, for any number of correlated stocks. In this context, I show that the replicating strategy amounts to nothing more than using the best rebalancing rule in hindsight over the currently known price history. The reported price matches the O(T^(n/2)) cost of universality derived by Cover in his “universal portfolio theory” (1986, 1991, 1996, 1998), which super-replicates the same derivative in discrete time. The replicating strategy compounds its money at the same asymptotic rate as the best levered rebalancing rule in hindsight, thereby beating the market asymptotically. Naturally enough, we find that American-style versions of this derivative will never be exercised early in equilibrium.

Hindsight Optimization for Hybrid State and Action MDPs

Hybrid (mixed discrete and continuous) state and action Markov Decision Processes (HSA-MDPs) provide an expressive formalism for modeling stochastic and concurrent sequential decision-making problems. Existing solvers for HSA-MDPs are either limited to very restricted transition distributions, require knowledge of domain-specific basis functions to achieve good approximations, or do not scale. We explore a domain-independent approach based on the framework of hindsight optimization (HOP) for HSA-MDPs, which uses an upper bound on the finite-horizon action values for action selection. Our main contribution is a linear time reduction to a Mixed Integer Linear Program (MILP) that encodes the HOP objective, when the dynamics are specified as location-scale probability distributions parametrized by Piecewise Linear (PWL) functions of states and actions. In addition, we show how to use the same machinery to select actions based on a lower-bound generated by straight line plans. Our empirical results show that the HSA-HOP approach effectively scales to high-dimensional problems and outperforms baselines that are capable of scaling to such large hybrid MDPs.

Shared Autonomy via Hindsight Optimization.

In shared autonomy, user input and robot autonomy are combined to control a robot to achieve a goal. Often, the robot does not know a priori which goal the user wants to achieve, and must both predict the user's intended goal, and assist in achieving that goal. We formulate the problem of shared autonomy as a Partially Observable Markov Decision Process with uncertainty over the user's goal. We utilize maximum entropy inverse optimal control to estimate a distribution over the user's goal based on the history of inputs. Ideally, the robot assists the user by solving for an action which minimizes the expected cost-to-go for the (unknown) goal. As solving the POMDP to select the optimal action is intractable, we use hindsight optimization to approximate the solution. In a user study, we compare our method to a standard predict-then-blend approach. We find that our method enables users to accomplish tasks more quickly while utilizing less input. However, when asked to rate each system, users were mixed in their assessment, citing a tradeoff between maintaining control authority and accomplishing tasks quickly.

Hindsight Optimization for Probabilistic Planning with Factored Actions

Inspired by the success of the satisfiability approach for deterministic planning, we propose a novel framework for on-line stochastic planning, by embedding the idea of hindsight optimization into a reduction to integer linear programming. In contrast to the previous work using reductions or hindsight optimization, our formulation is general purpose by working with domain specifications over factored state and action spaces, and by doing so is also scalable in principle to exponentially large action spaces. Our approach is competitive with state-of-the-art stochastic planners on challenging benchmark problems, and sometimes exceeds their performance especially in large action spaces.

Anticipatory On-Line Planning

We consider the problem of on-line continual planning, in whichadditional goals may arrive while plans for previous goals are stillexecuting and plan quality depends on how quickly goals are achieved.This is a challenging problem even in domains with deterministicactions. One common and straightforward approach is reactive planning,in which plans are synthesized when a new goal arrives. In this paper,we adapt the technique of hindsight optimization from on-line schedulingand probabilistic planning to create an anticipatory on-line planningalgorithm. Using an estimate of the goal arrival distribution, wesample possible futures and use a deterministic planner to estimate thevalue of taking possible actions at each time step. Results in twobenchmark domains based on unmanned aerial vehicle planning andmanufacturing suggest that an anticipatory approach yields a superiorplanner that is sensitive not only to which action should be executed,but when.

Approximate dynamic programming for an inventory problem: Empirical comparison

This study investigates the application of learning-based and simulation-based Approximate Dynamic Programming (ADP) approaches to an inventory problem under the Generalized Autoregressive Conditional Heteroscedasticity (GARCH) model. Specifically, we explore the robustness of a learning-based ADP method, Sarsa, with a GARCH(1,1) demand model, and provide empirical comparison between Sarsa and two simulation-based ADP methods: Rollout and Hindsight Optimization (HO). Our findings assuage a concern regarding the effect of GARCH(1,1) latent state variables on learning-based ADP and provide practical strategies to design an appropriate ADP method for inventory problems. In addition, we expose a relationship between ADP parameters and conservative behavior. Our empirical results are based on a variety of problem settings, including demand correlations, demand variances, and cost structures.

Lower Bounding Klondike Solitaire with Monte-Carlo Planning

Despite its ubiquitous presence, very little is known about the odds of winning the simple card game of Klondike Solitaire. The main goal of this paper is to investigate the use of probabilistic planning to shed light on this issue. Unfortunatley, most probabilistic planning techniques are not well suited for Klondike due to the difficulties of representing the domain in standard planning languages and the complexity of the required search. Klondike thus serves as an interesting addition to the complement of probabilistic planning domains. In this paper, we study Klondike using several sampling-based planning approaches including UCT, hindsight optimization, and sparse sampling, and establish lower bounds on their performance. We also introduce novel combinations of these approaches and evaluate them in Klondike. We provide a theoretical bound on the sample complexity of a method that naturally combines sparse sampling and UCT. Our results demonstrate that there is a policy that within tight confidence intervals wins over 35% of Klondike games. This result is the first reported lower bound of an optimal Klondike policy.

Burst-level congestion control using hindsight optimization

We consider the burst-level congestion-control problem in a communication network with multiple traffic sources, each modeled as a fully controllable stream of fluid traffic. The controlled traffic shares a common bottleneck node with high-priority cross traffic described by a Markov-modulated fluid (MMF). Each controlled source is assumed to have a unique round-trip delay. The goal is to maximize a linear combination of the throughput, delay, traffic-loss rate, and a fairness metric at the bottleneck node. We introduce a simulation-based congestion-control scheme capable of performing effectively under rapidly varying cross traffic by making use of the provided MMF model of that variation. The control problem is posed as a finite-horizon Markov decision process, and is solved heuristically using a technique called hindsight optimization. We provide a detailed derivation of our congestion-control algorithm based on this technique. Our empirical study shows that the control scheme performs significantly better than the conventional proportional-derivative congestion-control method.