Unknown Reward Research Articles

Minimax Optimal Bandits for Heavy Tail Rewards.

Stochastic multiarmed bandits (stochastic MABs) are a problem of sequential decision-making with noisy rewards, where an agent sequentially chooses actions under unknown reward distributions to minimize cumulative regret. The majority of prior works on stochastic MABs assume that the reward distribution of each action has bounded supports or follows light-tailed distribution, i.e., sub-Gaussian distribution. However, in a variety of decision-making problems, the reward distributions follow a heavy-tailed distribution. In this regard, we consider stochastic MABs with heavy-tailed rewards, whose p th moment is bounded by a constant νp for . First, we provide theoretical analysis on sub-optimality of the existing exploration methods for heavy-tailed rewards where it has been proven that existing exploration methods do not guarantee a minimax optimal regret bound. Second, to achieve the minimax optimality under heavy-tailed rewards, we propose a minimax optimal robust upper confidence bound (MR-UCB) by providing tight confidence bound of a p -robust estimator. Furthermore, we also propose a minimax optimal robust adaptively perturbed exploration (MR-APE) which is a randomized version of MR-UCB. In particular, unlike the existing robust exploration methods, both proposed methods have no dependence on νp . Third, we provide the gap-dependent and independent regret bounds of proposed methods and prove that both methods guarantee the minimax optimal regret bound for a heavy-tailed stochastic MAB problem. The proposed methods are the first algorithm that theoretically guarantees the minimax optimality under heavy-tailed reward settings to the best of our knowledge. Finally, we demonstrate the superiority of the proposed methods in simulation with Pareto and Fréchet noises with respect to regrets.

Online Learning Aided Decentralized Multi-User Task Offloading for Mobile Edge Computing

Mobile edge computing facilitates users to offload computation tasks to edge servers for meeting their stringent delay requirements. Previous works mainly explore task offloading when system-side information is given (e.g., server processing speed, cellular data rate), or centralized offloading under system uncertainty. But both generally fall short of handling task placement involving many coexisting users in an uncertain environment. In this paper, we develop a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">multi-user</i> offloading framework considering <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">unknown yet stochastic</i> system-side information to enable a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">decentralized user-initiated</i> service placement under overlapping server coverage. Specifically, we formulate the dynamic task placement as an online multi-user multi-armed bandit process, and propose a decentralized epoch based offloading (DEBO) to optimize user rewards which are subjected under network delay. We consider both cases without and with neighboring edge feedback once users' tasks are processed, where the latter incorporates system-side information sharing among edge servers for an enhanced task placement. For both cases, we show that DEBO can gradually deduce the optimal user-server assignment during dynamic offloading, thereby achieving a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">close-to-optimal</i> service performance and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">tight</i> <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$O(\log _{2} T)$</tex-math></inline-formula> regret. Moreover, we generalize DEBO to various common scenarios such as unknown reward gap, dynamic entering or leaving of clients, and fair reward distribution, while further exploring when users' offloaded tasks require <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">heterogeneous</i> computing resources. Particularly, we accomplish a sub-linear regret for each of these instances. Real measurements based evaluations corroborate the superiority of our offloading schemes over state-of-the-art approaches in optimizing delay-sensitive rewards.

Trajectory modeling via random utility inverse reinforcement learning

We consider the problem of modeling trajectories of drivers in a road network from the perspective of inverse reinforcement learning. Cars are detected by sensors placed on sparsely distributed points on the street network of a city. As rational agents, drivers are trying to maximize some reward function unknown to an external observer. We apply the concept of random utility from econometrics to model the unknown reward function as a function of observed and unobserved features. In contrast to current inverse reinforcement learning approaches, we do not assume that agents act according to a stochastic policy; rather, we assume that agents act according to a deterministic optimal policy and show that randomness in data arises because the exact rewards are not fully observed by an external observer. We introduce the concept of extended state to cope with unobserved features and develop a Markov decision process formulation of drivers decisions. We present theoretical results which guarantee the existence of solutions and show that maximum entropy inverse reinforcement learning is a particular case of our approach. Finally, we illustrate Bayesian inference on model parameters through a case study with real trajectory data from a large city in Brazil.

Motor System-Dependent Effects of Amygdala and Ventral Striatum Lesions on Explore-Exploit Behaviors.

Deciding whether to forego immediate rewards or explore new opportunities is a key component of flexible behavior and is critical for the survival of the species. Although previous studies have shown that different cortical and subcortical areas, including the amygdala and ventral striatum (VS), are implicated in representing the immediate (exploitative) and future (explorative) value of choices, the effect of the motor system used to make choices has not been examined. Here, we tested male rhesus macaques with amygdala or VS lesions on two versions of a three-arm bandit task where choices were registered with either a saccade or an arm movement. In both tasks we presented the monkeys with explore-exploit tradeoffs by periodically replacing familiar options with novel options that had unknown reward probabilities. We found that monkeys explored more with saccades but showed better learning with arm movements. VS lesions caused the monkeys to be more explorative with arm movements and less explorative with saccades, although this may have been due to an overall decrease in performance. VS lesions affected the monkeys' ability to learn novel stimulus-reward associations in both tasks, while after amygdala lesions this effect was stronger when choices were made with saccades. Further, on average, VS and amygdala lesions reduced the monkeys' ability to choose better options only when choices were made with a saccade. These results show that learning reward value associations to manage explore-exploit behaviors is motor system dependent and they further define the contributions of amygdala and VS to reinforcement learning.

Model-Free Reinforcement Learning for Mean Field Games

In this paper, we have proposed a model-free algorithm, based on sequential decomposition, to obtain optimal policies for. We consider finite horizon with a large population of homogeneous players, sequentially making strategic decisions. Each player observes a private state and a mean-field population state representing the empirical distribution of other players' states. The mean-field state is <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">common information</i> among all the players in the game. Authors in [1] provided a sequential decomposition algorithm to compute for such games in linear time than exponential as in prior literature. We extended the idea of sequential decomposition to propose a model-free algorithm for these games using Expected Sarsa in[2]. In this paper, we provide detailed convergence proofs for our algorithm. In addition, we propose an algorithm for with unknown reward functions. The proposed algorithm learns the reward function by studying an expert's behavior and then computes the optimal policy. We illustrate our results using a cyber-physical security example.

34 Neurocomputational Mechanisms of Social Reward Processing in Combat-Exposed Veterans

Objective:Combat exposure is associated with higher rates of depressive symptoms, including anhedonia (i.e., a reduced ability to seek and experience rewards) and feelings of social disconnectedness. While these symptoms are commonly documented in combat-exposed Veterans following deployment, the cognitive mechanisms underlying this pathology is less well understood. Computational modeling can provides detailed mechanistic insights into complex cognition, which may be particularly useful to understand how social reward processing is altered following combat exposure. Here, we use a Bayesian learning model framework to address this question.Participants and Methods:Thirty-three Operation Enduring Freedom (OEF)/ Operation Iraqi Freedom (OIF)/Operation New Dawn (OND) Veterans (25 Male, 8 Female) between the ages of 18-65 years old (M = 41.61, SD = 10.49) participated in this study. In both classic/monetary and social reward conditions, participants completed a 2-arm bandit task, in which they must choose on each trial between two options (i.e., slot machine vs social partner) with unknown reward rates. While they received monetary outcomes in the classic condition, participants received compliments from different fictitious partners in the social condition. We first compared a learning-independent Win-stay/Lose-shift (WSLS) heuristic and either a Rescorla-Wagner Q-learning or a Bayesian learning model (Dynamic Belief Model/DBM) paired with a Softmax reward maximization policy. DBM+Softmax provided the best fit of the data for most participants (31/33). Individual DBM parameters of prior reward expectation, reward learning (i.e., perceived stability of reward rates), and Softmax reward maximization were estimated and compared across conditions.Results:Participants did not differ in their reward learning parameters across monetary and social conditions (t(30)= -0.70, p = 0.490), suggesting similar perception of reward stability in both modalities. However, higher Bayesian prior mean (i.e., initial belief of reward rate; t(30)= -2.31, p = 0.028, d=0.42) and greater reward maximization (i.e., Softmax parameter; t(30)= -2.26, p = 0.031, d=0.41) were observed in response to social vs monetary rewards. In the social reward condition, higher self-reported social connectedness was associated with greater model fit of our DBM model (i.e., smaller Bayesian Information Criterion/BIC; r = -0.38, p = 0.041). In this condition, those expecting higher reward rates when initiating reward exploration (those with higher DBM prior mean) endorsed lower self-esteem (Spearman's ρ = -0.43, p = 0.078) and lower positive affect (ρ = -0.32, p = 0.078).Conclusions:A Bayesian learning modeling framework can characterize mechanistic differences in the processing of social vs non-social reward among combat-exposed Veterans. Individuals with higher social connectedness were more model-based in their performance, consistent with the notion that they are more likely to estimate and anticipate how much social peers have to offer. Combat-exposed individuals with lower self-esteem and positive affect appear to have higher initial expectations of reward from unknown partners, which could reflect greater need for mood and/or self-esteem repair in those individuals. Overall, Bayesian modeling of social reward behavior provides a useful quantitative framework to predict clinically relevant construct of functional outcomes in military populations.

Reward-Biased Maximum Likelihood Estimation for Neural Contextual Bandits: A Distributional Learning Perspective

Reward-biased maximum likelihood estimation (RBMLE) is a classic principle in the adaptive control literature for tackling explore-exploit trade-offs. This paper studies the neural contextual bandit problem from a distributional perspective and proposes NeuralRBMLE, which leverages the likelihood of surrogate parametric distributions to learn the unknown reward distributions and thereafter adapts the RBMLE principle to achieve efficient exploration by properly adding a reward-bias term. NeuralRBMLE leverages the representation power of neural networks and directly encodes exploratory behavior in the parameter space, without constructing confidence intervals of the estimated rewards. We propose two variants of NeuralRBMLE algorithms: The first variant directly obtains the RBMLE estimator by gradient ascent, and the second variant simplifies RBMLE to a simple index policy through an approximation. We show that both algorithms achieve order-optimality. Through extensive experiments, we demonstrate that the NeuralRBMLE algorithms achieve comparable or better empirical regrets than the state-of-the-art methods on real-world datasets with non-linear reward functions.

Order recognition by Schubert polynomials generated by optical near-field statistics via nanometre-scale photochromism

Irregular spatial distribution of photon transmission through a photochromic crystal photoisomerized by a local optical near-field excitation was previously reported, which manifested complex branching processes via the interplay of material deformation and near-field photon transfer therein. Furthermore, by combining such naturally constructed complex photon transmission with a simple photon detection protocol, Schubert polynomials, the foundation of versatile permutation operations in mathematics, have been generated. In this study, we demonstrated an order recognition algorithm inspired by Schubert calculus using optical near-field statistics via nanometre-scale photochromism. More specifically, by utilizing Schubert polynomials generated via optical near-field patterns, we showed that the order of slot machines with initially unknown reward probability was successfully recognized. We emphasized that, unlike conventional algorithms, the proposed principle does not estimate the reward probabilities but exploits the inversion relations contained in the Schubert polynomials. To quantitatively evaluate the impact of Schubert polynomials generated from an optical near-field pattern, order recognition performances were compared with uniformly distributed and spatially strongly skewed probability distributions, where the optical near-field pattern outperformed the others. We found that the number of singularities contained in Schubert polynomials and that of the given problem or considered environment exhibited a clear correspondence, indicating that superior order recognition is attained when the singularity of the given situations is presupposed. This study paves way for physical computing through the interplay of complex natural processes and mathematical insights gained by Schubert calculus.

Probably Approximately Correct Learning in Adversarial Environments With Temporal Logic Specifications

Reinforcement learning (RL) algorithms have been used to learn how to implement tasks in uncertain and partially unknown environments. In practice, environments are usually uncontrolled and may affect task performance in an adversarial way. In this article, we model the interaction between an RL agent and its potentially adversarial environment as a turn-based zero-sum stochastic game. The task requirements are represented both qualitatively as a subset of linear temporal logic (LTL) specifications, and quantitatively as a reward function. For each case in which the LTL specification is realizable and can be equivalently transformed into a deterministic Büchi automaton, we show that there always exists a memoryless almost-sure winning strategy that is <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\varepsilon$</tex-math></inline-formula> -optimal for the discounted-sum objective for any arbitrary positive <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\varepsilon$</tex-math></inline-formula> . We propose a probably approximately correct (PAC) learning algorithm that learns such a strategy efficiently in an online manner with <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">a priori</i> unknown reward functions and unknown transition distributions. To the best of our knowledge, this is the first result on PAC learning in stochastic games with independent quantitative and qualitative objectives.

Chasing Unknown Bandits: Uncertainty Guidance in Learning and Decision Making

In repeated decision problems for which it is possible to learn from experience, people should actively seek out uncertain options, rather than avoid ambiguity or uncertainty, in order to learn and improve future decisions. Research on human behavior in a variety of multiarmed-bandit tasks supports this prediction. Multiarmed-bandit tasks involve repeated decisions between options with initially unknown reward distributions and require a careful balance between learning about relatively unknown options (exploration) and obtaining high immediate rewards (exploitation). Resolving this exploration-exploitation dilemma optimally requires considering not only the estimated value of each option, but also the uncertainty in these estimations. Bayesian learning naturally quantifies uncertainty and hence provides a principled framework to study how humans resolve this dilemma. On the basis of computational modeling and behavioral results in bandit tasks, I argue that human learning, attention, and exploration are guided by uncertainty. These results support Bayesian theories of cognition and underpin the fundamental role of subjective uncertainty in both learning and decision making.

Model-based inverse reinforcement learning for deterministic systems

This paper focuses on the development of an online data-driven model-based inverse reinforcement learning (MBIRL) technique for linear and nonlinear deterministic systems. Input and output trajectories of an agent under observation, attempting to optimize an unknown reward function, are used to estimate the reward function and the corresponding unknown optimal value function, online and in real-time. To achieve MBIRL using limited data, a novel feedback-driven approach to MBIRL is developed. The feedback policy and the dynamic model of the agent under observation are estimated from the measured data and the estimates are used to generate synthetic data to drive MBIRL. Theoretical guarantees for ultimate boundedness of the estimation errors in general, and convergence of the estimation errors to zero in special cases, are derived using Lyapunov techniques. Proof of concept numerical experiments demonstrates the utility of the developed method to solve linear and nonlinear inverse reinforcement learning problems.

Decision-Making Processes Related to Perseveration Are Indirectly Associated With Weight Status in Children Through Laboratory-Assessed Energy Intake.

Decision-making contributes to what and how much we consume, and deficits in decision-making have been associated with increased weight status in children. Nevertheless, the relationships between cognitive and affective processes underlying decision-making (i.e., decision-making processes) and laboratory food intake are unclear. We used data from a four-session, within-subjects laboratory study to investigate the relationships between decision-making processes, food intake, and weight status in 70 children 7-to-11-years-old. Decision-making was assessed with the Hungry Donkey Task (HDT), a child-friendly task where children make selections with unknown reward outcomes. Food intake was measured with three paradigms: (1) a standard ad libitum meal, (2) an eating in the absence of hunger (EAH) protocol, and (3) a palatable buffet meal. Individual differences related to decision-making processes during the HDT were quantified with a reinforcement learning model. Path analyses were used to test whether decision-making processes that contribute to children’s (a) expected value of a choice and (b) tendency to perseverate (i.e., repeatedly make the same choice) were indirectly associated with weight status through their effects on intake (kcal). Results revealed that increases in the tendency to perseverate after a gain outcome were positively associated with intake at all three paradigms and indirectly associated with higher weight status through intake at both the standard and buffet meals. Increases in the tendency to perseverate after a loss outcome were positively associated with EAH, but only in children whose tendency to perseverate persistedacross trials. Results suggest that decision-making processes that shape children’s tendencies to repeat a behavior (i.e., perseverate) are related to laboratory energy intake across multiple eating paradigms. Children who are more likely to repeat a choice after a positive outcome have a tendency to eat more at laboratory meals. If this generalizes to contexts outside the laboratory, these children may be susceptible to obesity. By using a reinforcement learning model not previously applied to the study of eating behaviors, this study elucidated potential determinants of excess energy intake in children, which may be useful for the development of childhood obesity interventions.

Photonic decision-making for arbitrary-number-armed bandit problem utilizing parallel chaos generation

In this paper, we propose and experimentally demonstrate a novel scheme that helps to solve an any-number-armed bandit problem by utilizing two parallel simultaneously-generated chaotic signals and the epsilon (ɛ)-greedy strategy. In the proposed scheme, two chaotic signals are experimentally generated, and then processed by an 8-bit analog-to-digital conversion (ADC) with 4 least significant bits (LSBs), to generate two amplitude-distribution-uniform sequences for decision-making. The correspondence between these two random sequences and different arms is established by a mapping rule designed in virtue of the ɛ-greedy-strategy. Based on this, decision-making for an exemplary 5-armed bandit problem is successfully performed, and moreover, the influences of the mapping rule and unknown reward probabilities on the correction decision rate (CDR) performance for the 4-armed to 7-armed bandit problems are investigated. This work provides a novel way for solving the arbitrary-number-armed bandit problem.

Dynamic multiobjective optimization driven by inverse reinforcement learning

Due to the widespread interest in dynamic multiobjective optimization in real-world applications, more and more approaches exploiting machine learning are deployed to tackle this type of problems. Unfortunately, recent works do not make full use of the data obtained during the optimization process, which could be benefit for model training thereby mining the dynamic characteristics of the underlying problem. To address this issue, this paper proposes a dynamic multiobjective evolutionary algorithm driven by inverse reinforcement learning to solve the dynamic multiobjective optimization problems. IRL is widely used to recover the unknown reward function, making it possible to perform at an expert level. The notable features of the proposed algorithm mainly consist of data-driven evolutionary technique, which uses inverse reinforcement learning as a surrogate-assisted model for model training. This design makes full use of the surrogate management strategy based on inverse reinforcement learning to optimize the reward function within a reinforcement learning framework. At the same time, the algorithm can generate a promising policy based on limited training data during the optimization process to achieve better algorithm evolution and guide the search. The experimental results on the benchmark problems validate that the proposed algorithm is effective in dealing with dynamic multiobjective optimization.

Adaptive Algorithms for Multi-armed Bandit with Composite and Anonymous Feedback

We study the multi-armed bandit (MAB) problem with composite and anonymous feedback. In this model, the reward of pulling an arm spreads over a period of time (we call this period as reward interval) and the player receives partial rewards of the action, convoluted with rewards from pulling other arms, successively. Existing results on this model require prior knowledge about the reward interval size as an input to their algorithms. In this paper, we propose adaptive algorithms for both the stochastic and the adversarial cases, without requiring any prior information about the reward interval. For the stochastic case, we prove that our algorithm guarantees a regret that matches the lower bounds (in order). For the adversarial case, we propose the first algorithm to jointly handle non-oblivious adversary and unknown reward interval size. We also conduct simulations based on real-world dataset. The results show that our algorithms outperform existing benchmarks.

Reinforcement Learning with Trajectory Feedback

The standard feedback model of reinforcement learning requires revealing the reward of every visited state-action pair. However, in practice, it is often the case that such frequent feedback is not available. In this work, we take a first step towards relaxing this assumption and require a weaker form of feedback, which we refer to as \emph{trajectory feedback}. Instead of observing the reward obtained after every action, we assume we only receive a score that represents the quality of the whole trajectory observed by the agent, namely, the sum of all rewards obtained over this trajectory. We extend reinforcement learning algorithms to this setting, based on least-squares estimation of the unknown reward, for both the known and unknown transition model cases, and study the performance of these algorithms by analyzing their regret. For cases where the transition model is unknown, we offer a hybrid optimistic-Thompson Sampling approach that results in a tractable algorithm.

Modeling pedestrian-cyclist interactions in shared space using inverse reinforcement learning

The objective of this study is to model the microscopic behaviour of mixed traffic (cyclist-pedestrian) interactions in non-motorized shared spaces. Video data were collected at two locations of Robson Square non-motorized shared space in downtown Vancouver, British Columbia. Trajectories of cyclists and pedestrians involved in interactions were extracted using computer vision algorithms. The extracted trajectories were used to obtain several variables that describe elements of road users’ behaviour including longitudinal and lateral distances, speed and speed differences, interaction angle, and cyclist acceleration and yaw rate. The road users behaviour was modeled as utility-based intelligent rational agents using the finite-state Markov Decision Process (MDP) framework with unknown reward functions. The study implemented Inverse Reinforcement Learning (IRL) using two algorithms: the Maximum Entropy (ME) algorithm, and the Feature Matching (FM) algorithm to recover/estimate the reward function weights of cyclists in two types of interactions with pedestrians: following and overtaking interactions. Reward function weights infer cyclist preferences during their interactions with pedestrians in non-motorized shared spaces, and can form the key component in developing agent based microsimulation model for road users. Furthermore, the estimated reward functions were used to estimate cyclists’ optimal policy for such interactions. A simulation platform was developed using the estimated reward functions and the cyclist optimal policies to simulate cyclist trajectories for the validation dataset. Results show that the Maximum Entropy (ME) IRL algorithm outperformed the Feature Matching (FM) IRL algorithm, and generally provided reasonable results for modeling such interactions in non-motorized shared spaces, considering the high degrees of freedom in movement and the more-complex road users interactions in such facilities. This research is considered an important step toward developing a full Agent-Based Model (ABM) for road users in shared space facilities to evaluate the safety and efficiency of such facilities.

An end-to-end inverse reinforcement learning by a boosting approach with relative entropy

Inverse reinforcement learning (IRL) involves imitating expert behaviors by recovering reward functions from demonstrations. This study proposes a model-free IRL algorithm to solve the dilemma of predicting the unknown reward function. The proposed end-to-end model comprises a dual structure of autoencoders in parallel. The model uses a state encoding method to reduce the computational complexity for high-dimensional environments and utilizes an Adaboost classifier to determine the difference between the predicted and demonstrated reward functions. Relative entropy is used as a metric to measure the difference between the demonstrated and the imitated behavior. The simulation experiments demonstrate the effectiveness of the proposed method in terms of the number of iterations that are required for the estimation.

Information Constrained Control Analysis of Eye Gaze Distribution Under Workload

We describe a novel model of human eye gaze behavior under workload, derived from the basic principle of information constrained control. The model assumes two distributions over the visual field: A saliency distribution, which is nongoal oriented, and a reward task-related distribution. The eye gaze behavior is determined by the tradeoff between these two distributions, where the goal is to preserve the task-related constraints, while remaining as close as possible to the saliency distribution representing a comfort zone. Based on minimum Kullback–Liebler divergence principles, the model gives rise to a family of gaze distributions controlled by a single tradeoff parameter. The model was evaluated experimentally in a driving simulator that consisted of an immersive environment with clear tasks and accurate monitoring capabilities. The findings confirm the theoretical predictions with respect to the low rank manifold and order relations in the data. We show that the model can be used to visualize the unknown reward function associated with a task, and predict human workload based on gaze pattern.

Optimal Exploration–Exploitation in a Multi-armed Bandit Problem with Non-stationary Rewards

In a multi-armed bandit problem, a gambler needs to choose at each round one of K arms, each characterized by an unknown reward distribution. The objective is to maximize cumulative expected earnings over a planning horizon of length T, and performance is measured in terms of regret relative to a (static) oracle that knows the identity of the best arm a priori. This problem has been studied extensively when the reward distributions do not change over time, and uncertainty essentially amounts to identifying the optimal arm. We complement this literature by developing a flexible non-parametric model for temporal uncertainty in the rewards. The extent of temporal uncertainty is measured via the cumulative mean change in the rewards over the horizon, a metric we refer to as temporal variation, and regret is measured relative to a (dynamic) oracle that plays the point-wise optimal action at each period. Assuming that nature can choose any sequence of mean rewards such that their temporal variation does not exceed V (a temporal uncertainty budget), we characterize the complexity of this problem via the minimax regret, which depends on V (the hardness of the problem), the horizon length T, and the number of arms K.