Constant Regret Research Articles

Bandits with Stochastic Experts: Constant Regret, Empirical Experts and Episodes

We study a variant of the contextual bandit problem where an agent can intervene through a set of stochastic expert policies. Given a fixed context, each expert samples actions from a fixed conditional distribution. The agent seeks to remain competitive with the “best” among the given set of experts. We propose the Divergence-based Upper Confidence Bound (D-UCB) algorithm that uses importance sampling to share information across experts and provide horizon-independent constant regret bounds that only scale linearly in the number of experts. We also provide the Empirical D-UCB (ED-UCB) algorithm that can function with only approximate knowledge of expert distributions. Further, we investigate the episodic setting where the agent interacts with an environment that changes over episodes. Each episode can have different context and reward distributions resulting in the best expert changing across episodes. We show that by bootstrapping from \(\mathcal {O}(N\log (NT^2\sqrt {E}))\) samples, ED-UCB guarantees a regret that scales as \(\mathcal {O}(E(N+1) + \frac{N\sqrt {E}}{T^2})\) for N experts over E episodes, each of length T . We finally empirically validate our findings through simulations.

Online Markov Decision Processes Configuration with Continuous Decision Space

In this paper, we investigate the optimal online configuration of episodic Markov decision processes when the space of the possible configurations is continuous. Specifically, we study the interaction between a learner (referred to as the configurator) and an agent with a fixed, unknown policy, when the learner aims to minimize her losses by choosing transition functions in online fashion. The losses may be unrelated to the agent's rewards. This problem applies to many real-world scenarios where the learner seeks to manipulate the Markov decision process to her advantage. We study both deterministic and stochastic settings, where the losses are either fixed or sampled from an unknown probability distribution. We design two algorithms whose peculiarity is to rely on occupancy measures to explore with optimism the continuous space of transition functions, achieving constant regret in deterministic settings and sublinear regret in stochastic settings, respectively. Moreover, we prove that the regret bound is tight with respect to any constant factor in deterministic settings. Finally, we compare the empiric performance of our algorithms with a baseline in synthetic experiments.

Efficient Learning in Polyhedral Games via Best-Response Oracles

We study online learning and equilibrium computation in games with polyhedral decision sets, a property shared by normal-form games (NFGs) and extensive-form games (EFGs), when the learning agent is restricted to utilizing a best-response oracle. We show how to achieve constant regret in zero-sum games and O(T^0.25) regret in general-sum games while using only O(log t) best-response queries at a given iteration t, thus improving over the best prior result, which required O(T) queries per iteration. Moreover, our framework yields the first last-iterate convergence guarantees for self-play with best-response oracles in zero-sum games. This convergence occurs at a linear rate, though with a condition-number dependence. We go on to show a O(T^(-0.5)) best-iterate convergence rate without such a dependence. Our results build on linear-rate convergence results for variants of the Frank-Wolfe (FW) algorithm for strongly convex and smooth minimization problems over polyhedral domains. These FW results depend on a condition number of the polytope, known as facial distance. In order to enable application to settings such as EFGs, we show two broad new results: 1) the facial distance for polytopes in standard form is at least γ/k where γ is the minimum value of a nonzero coordinate of a vertex of the polytope and k≤n is the number of tight inequality constraints in the optimal face, and 2) the facial distance for polytopes of the form Ax=b, Cx≤d, x≥0 where x∈R^n, C≥0 is a nonzero integral matrix, and d≥0, is at least 1/(c√n), where c is the infinity norm of C. This yields the first such results for several problems such as sequence-form polytopes, flow polytopes, and matching polytopes.

Distributed online optimization for heterogeneous linear multi-agent systems with coupled constraints

This paper studies a class of distributed online convex optimization problems for heterogeneous linear multi-agent systems. Agents in a network, knowing only their own outputs, need to minimize the time-varying costs through neighboring interaction subject to time-varying coupled inequality constraints. Based on the saddle-point technique, we design a continuous-time distributed controller which is shown to achieve constant regret bound and sublinear fit bound, matching those of the standard centralized online method. We further extend the control law to the event-triggered communication mechanism and show that the constant regret bound and sublinear fit bound are still achieved while reducing the communication frequency. Additionally, we study the situation of communication noise, i.e., the agent’s measurement of the relative states of its neighbors is disturbed by a noise. It is shown that, if the noise is not excessive, the regret and fit bounds are unaffected, which indicates the controller’s noise-tolerance capability to some extent. Finally, a numerical simulation is provided to support the theoretical conclusions.

On the regret of online edge service hosting

We consider the problem of service hosting where a service provider can dynamically rent edge resources via short term contracts to ensure better quality of service to its customers. The service can also be partially hosted at the edge, in which case, customers’ requests can be partially served at the edge. The total cost incurred by the system is modeled as a combination of the rent cost, the service cost incurred due to latency in serving customers, and the fetch cost incurred as a result of the bandwidth used to fetch the code/databases of the service from the cloud servers to host the service at the edge. In this paper, we compare multiple hosting policies with regret as a metric, defined as the difference in the cost incurred by the policy and the optimal policy over some time horizon T. In particular we consider the Retro Renting (RR) and Follow The Perturbed Leader (FTPL) policies proposed in the literature and provide performance guarantees on the regret of these policies. We show that under i.i.d stochastic arrivals, RR policy has linear regret while FTPL policy has constant regret. Next, we propose a variant of FTPL, namely Wait then FTPL (W-FTPL), which also has constant regret while demonstrating much better dependence on the fetch cost. We also show that under adversarial arrivals, RR policy has linear regret while both FTPL and W-FTPL have regret O(T) which is order-optimal.

Dynamic Matching: Characterizing and Achieving Constant Regret

We study how to optimally match agents in a dynamic matching market with heterogeneous match cardinalities and values. A network topology determines the feasible matches in the market. In general, a fundamental tradeoff exists between short-term value—which calls for performing matches frequently—and long-term value—which calls, sometimes, for delaying match decisions in order to perform better matches. We find that in networks that satisfy a general position condition, the tension between short- and long-term value is limited, and a simple periodic clearing policy (nearly) maximizes the total match value simultaneously at all times. Central to our results is the general position gap ϵ; a proxy for capacity slack in the market. With the exception of trivial cases, no policy can achieve an all-time regret that is smaller, in terms of order, than [Formula: see text]. We achieve this lower bound with a policy, which periodically resolves a natural matching integer linear program, provided that the delay between resolving periods is of the order of [Formula: see text]. Examples illustrate the necessity of some delay to alleviate the tension between short- and long-term value. This paper was accepted by David Simchi-Levi, revenue management and market analytics. Funding: This work was supported by the National Science Foundation [Grant CMM-2010940] and the U.S. Department of Defense [Grant STTR A18B-T007].

On the Regret of Online Edge Service Hosting

We consider the problem of service hosting where a service provider can dynamically rent edge resources via short term contracts to ensure better quality of service to its customers. The total cost incurred by the system is modeled as a combination of the rent cost, the service cost incurred due to latency in serving customers, and the fetch cost incurred as a result of the bandwidth used to fetch the code/databases of the service from the cloud servers to host the service at the edge. In this paper, we compare multiple hosting policies with regret as a metric, defined as the difference in the cost incurred by the policy and the optimal policy over some time horizon T. In particular we consider the Retro Renting (RR) and Follow The Perturbed Leader (FTPL) policies proposed in the literature and provide performance guarantees on the regret of these policies. We show that under i.i.d Bernoulli arrivals, RR policy has linear regret while FTPL policy has constant regret. Next, we propose a variant of FTPL, namely Wait then FTPL (W-FTPL), which also has constant regret while demonstrating much better dependence on the fetch cost. We also show that under adversarial arrivals, RR policy has linear regret while both FTPL and W-FTPL have regret O(pT) which is order-optimal.

Bayesian dithering for learning: Asymptotically optimal policies in dynamic pricing

We consider a dynamic pricing and learning problem where a seller prices multiple products and learns from sales data about unknown demand. We study the parametric demand model in a Bayesian setting. To avoid the classical problem of incomplete learning, we propose dithering policies under which prices are probabilistically selected in a neighborhood surrounding the myopic optimal price. By analyzing the effect of dithering in facilitating learning, we establish regret upper bounds for three typical settings of demand model. We show that the dithering policy achieves an upper bound of order [Formula: see text] when the parameter set is finite. It can be modified to achieve a constant regret bound under an additional assumption. We also prove an upper bound of order [Formula: see text] when the parameter set is compact and convex. Each bound matches (up to a logarithmic factor) the existing lower bound of any pricing policy. In this way, we show that dithering policies achieve asymptotically optimal performance in three different parameter settings, which demonstrates dithering as a unified approach to strike the balance between exploration and exploitation.

Constant Regret Resolving Heuristics for Price-Based Revenue Management

Title: Constant Regret Resolving Heuristics for Price-Based Revenue Management Network revenue management (NRM) and its corresponding pricing question is one of the most fundamental problems in operations management. To alleviate the curse of dimensionality and the prohibitive cost of computing an exact solution using dynamic programming, computationally efficient resolving algorithms are proposed. The state-of-the-art analysis of the resolving heuristic establishes a logarithmic additive regret for price-based NRM problems. In “Constant Regret Resolving Heuristics for Price-Based Revenue Management,” Y. Wang and H. Wang from the University of Florida and Georgia Institute of Technology, respectively, significantly advance the state-of-the-art analysis by showing a constant regret for resolving heuristics. Their theoretical advance is made possible by a novel, direct analysis of the exact DP solution.

One for All and All for One: Distributed Learning of Fair Allocations With Multi-Player Bandits

Consider N cooperative but non-communicating players where each plays one out of M arms for T turns. Players have different utilities for each arm, represented as an N×M matrix. These utilities are unknown to the players. In each turn, players select an arm and receive a noisy observation of their utility for it. However, if any other players selected the same arm in that turn, all colliding players will receive zero utility due to the conflict. No communication between the players is possible. We propose two distributed algorithms which learn fair matchings between players and arms while minimizing the regret. We show that our first algorithm learns a max-min fairness matching with near- O(logT) regret (up to a loglogT factor). However, if one has a known target Quality of Service (QoS) (which may vary between players) then we show that our second algorithm learns a matching where all players obtain an expected reward of at least their QoS with constant regret, given that such a matching exists. In particular, if the max-min value is known, a max-min fairness matching can be learned with O(1) regret.

Policy Optimization as Online Learning with Mediator Feedback

Policy Optimization (PO) is a widely used approach to address continuous control tasks. In this paper, we introduce the notion of mediator feedback that frames PO as an online learning problem over the policy space. The additional available information, compared to the standard bandit feedback, allows reusing samples generated by one policy to estimate the performance of other policies. Based on this observation, we propose an algorithm, RANDomized-exploration policy Optimization via Multiple Importance Sampling with Truncation (RANDOMIST), for regret minimization in PO, that employs a randomized exploration strategy, differently from the existing optimistic approaches. When the policy space is finite, we show that under certain circumstances, it is possible to achieve constant regret, while always enjoying logarithmic regret. We also derive problem-dependent regret lower bounds. Then, we extend RANDOMIST to compact policy spaces. Finally, we provide numerical simulations on finite and compact policy spaces, in comparison with PO and bandit baselines.

Multiplayer Bandits: A Trekking Approach

We study stochastic multi-armed bandits with many players. The players do not know the number of players, cannot communicate with each other and if multiple players select a common arm they collide and none of them receive any reward. We consider the static scenario, where the number of players remains fixed, and the dynamic scenario, where the players enter and leave at any time. We provide algorithms based on a novel `trekking approach' that guarantees constant regret for the static case and sub-linear regret for the dynamic case with high probability. The trekking approach eliminates the need to estimate the number of players resulting in fewer collisions and improved regret performance compared to the state-of-the-art algorithms. We also develop an epoch-less algorithm that eliminates any requirement of time synchronization across the players provided each player can detect the presence of other players on an arm. We validate our theoretical guarantees using simulation based and real test-bed based experiments.

Online Allocation and Pricing: Constant Regret via Bellman Inequalities

We develop a framework for designing simple and efficient policies for a family of online allocation and pricing problems that includes online packing, budget-constrained probing, dynamic pricing, and online contextual bandits with knapsacks. In each case, we evaluate the performance of our policies in terms of their regret (i.e., additive gap) relative to an offline controller that is endowed with more information than the online controller. Our framework is based on Bellman inequalities, which decompose the loss of an algorithm into two distinct sources of error: (1) arising from computational tractability issues, and (2) arising from estimation/prediction of random trajectories. Balancing these errors guides the choice of benchmarks, and leads to policies that are both tractable and have strong performance guarantees. In particular, in all our examples, we demonstrate constant-regret policies that only require resolving a linear program in each period, followed by a simple greedy action-selection rule; thus, our policies are practical as well as provably near optimal.

Dynamic Matching: Characterizing and Achieving Constant Regret

We study how to optimally match agents in a dynamic market with heterogeneous match values. A network topology determines the feasible matches in the market. We consider networks that are two-sided when all matches include two agents, or acyclic otherwise. An inherent trade-off arises between generating short- and long-term value. We find that when the network satisfies a general position condition, this trade-off is limited, and a simple periodic clearing policy (nearly) maximizes the total value simultaneously at all times. Central to our results is the general position gap, e, which quantifies the stability or the imbalance in the network. No policy can achieve a regret that is lower than the order of 1/e at all times. This lower bound is achieved by a policy, which periodically resolves a natural LP, provided that the delay between periods is of the order of 1/e. Examples illustrate the necessity of some delay in periodic clearing policies.

Second-Order Online Nonconvex Optimization

We present the online Newton's method, a single-step second-order method for online nonconvex optimization. We analyze its performance and obtain a dynamic regret bound that is linear in the cumulative variation between round optima. We show that if the variation between round optima is limited, the method leads to a constant regret bound. In the general case, the online Newton's method outperforms online convex optimization algorithms for convex functions and performs similarly to a specialized algorithm for strongly convex functions. We simulate the performance of the online Newton's method on a nonlinear, nonconvex moving target localization example and find that it outperforms a first-order approach.

Dithering for Learning: Computationally Efficient Policies for Dynamic Pricing in High Dimensions

We consider a dynamic pricing and learning problem where a seller prices multiple products and learns from sales data about unknown demand. We propose dithering policies---namely, policies under which prices are probabilistically selected in a neighborhood surrounding the myopic optimal price---that achieve good theoretical performance and are computationally efficient. In particular, by analyzing structural properties of price-dithering, we establish a regret bound of order √T logT. Moreover, we prove under an additional assumption that our policy can be modified to achieve a constant regret bound. With regard to computation, we show that our dithering policies are as efficient as the widely-used myopic policy, yet avoid the problem of incomplete learning that can be encountered under the myopic policy. Dithering, which can be viewed as a soft or probabilistic avoidance of incomplete learning, is also shown to be substantially more computationally efficient than existing discriminative policies which impose (typically non-convex) hard constraints on the price space and can often be prohibitively time consuming to solve at the optimization stage, particularly in high dimensions.

Distributed Learning and Coordination in Cognitive Infrastructureless Networks of Unknown Size

The cognitive infrastructureless network has received significant attention to reduce signaling load in next-generation networks, which are expected to be ultradense with very high peak rate but relatively lower expected traffic per user. Research in this area has evolved from the distributed algorithms requiring prior knowledge of the number of secondary users (SUs) $U$ to the existing algorithms, which can estimate $U$ independently by counting the number of collisions. The major drawback of these algorithms is the large number of collisions leading to wastage of power and bring down the effective life of battery operated SUs. In this paper, we develop algorithms that learn faster and incurs fewer collisions. We assume unknown $U$ with two types of networks: fixed $U$ (i.e., static networks) and time-varying $U$ (i.e., dynamic networks). Proposed algorithms are based on the multiplayer multi-armed bandit (MAB) approach. We show that the proposed algorithms offer constant regret (i.e., throughput loss) with high probability and sub-linear regret in static and dynamic networks, respectively. Theoretical analysis, simulation, and experimental results demonstrate the efficacy of the proposed algorithms in terms of vacant spectrum utilization, regret, and the number of collisions. Fewer collisions significantly increase the operational life of SUs.

Distributed Learning in Ad-Hoc Networks with Unknown Number of Players

We study algorithms for distributed learning in ad-hoc cognitive networks where no central controller is available. In such networks, the players cannot communicate with each other and even may not know how many other players are present in the network. If multiple players select a common channel they collide, which results in loss of throughput for the colliding players. We consider both the static and dynamic scenarios where the number of players remains fixed throughout the game in the former case and can change in the later. We provide algorithms based on a novel 'trekking approach' that guarantees with high probability constant regret for the static case and sub-linear regret for the dynamic case. The trekking approach gives improved aggregate throughput and also results in fewer collisions compared to the state-of-the-art algorithms.

Satisfactory content delivery scheme for QoS provisioning in delay tolerant networks

SummaryWe deal in this article with the content forwarding problem in delay tolerant networks (DTNs). We first formulate the content delivery interaction as a noncooperative satisfaction game. On one hand, the source node seeks to ensure a delivery probability above some given threshold. On the other hand, the relay nodes seek to maximize their own payoffs. The source node offers a reward (virtual coins) to the relay, which caches and forwards the file to the final destination. Each relay has to solve the dilemma of accepting/rejecting to cache the source's file. Cooperation incurs energy cost due to caching, carrying, and forwarding the source's file. Yet when a relay accepts to cooperate, it may receive some reward if it succeeds to be the first relay to forward the content to the destination. Otherwise, the relay may receive some penalty in the form of a constant regret; the latter parameter is introduced to make incentive for cooperation. Next, we introduce the concept of satisfaction equilibrium (SE) as a solution concept to the induced game. Now, the source node is solely interested in reaching a file delivery probability greater than some given threshold, while the relays behave rationally to maximize their respective payoffs. Full characterizations of the SEs for both pure and mixed strategies are derived. Furthermore, we propose two learning algorithms allowing the players (source/relays) to reach the SE strategies. Finally, extensive numerical investigations and some learning simulations are carried out to illustrate the behavior of the interacting nodes and to give some insightful thoughts on how to fine‐tune the network setting.

Online Learning Schemes for Power Allocation in Energy Harvesting Communications

We consider the problem of power allocation over one or more time-varying channels with unknown distributions in energy harvesting communications. In the single-channel case, the transmitter chooses the transmit power based on the amount of stored energy in its battery with the goal of maximizing the average rate over time. We model this problem as a Markov decision process (MDP) with transmitter as the agent, battery status as the state, transmits power as the action and rate as the reward. The average reward maximization problem can be modeled by a linear program (LP) that uses the transition probabilities for the state-action pairs and their reward values to select a power allocation policy. This problem is challenging because the uncertainty in channels implies that the mean rewards associated with the state-action pairs are unknown. We therefore propose two online learning algorithms: linear program of sample means (LPSM) and Epoch-LPSM that learn these rewards and adapt their policies over time. For both algorithms, we prove that their regret is upper-bounded by a constant. To our knowledge this is the first result showing constant regret learning algorithms for MDPs with unknown mean rewards. We also prove an even stronger result about LPSM: that its policy matches the optimal policy exactly in finite expected time. Epoch-LPSM incurs a higher regret compared with the LPSM, while reducing the computational requirements substantially. We further consider a multi-channel scenario, where the agent also chooses a channel in each slot, and present our multi-channel LPSM (MC-LPSM) algorithm that explores different channels and uses that information to solve the LP during exploitation. MC-LPSM incurs a regret that scales logarithmically in time and linearly in the number of channels. Through a matching lower bound on the regret of any algorithm, we also prove the asymptotic order optimality of MC-LPSM.