Bandit Feedback Research Articles

Learning the Optimal Control for Evolving Systems with Converging Dynamics

We consider a principle or controller that can pick actions from a fixed action set to control an evolving system with converging dynamics. The converging dynamics means that, if the principle holds the same action, the system will asymptotically converge to a unique stable state determined by this action. In our model, the dynamics of the system are unknown to the principle, and the principle can only receive bandit feedback (maybe noisy) on the impacts of his actions. The principle aims to learn which stable state yields the highest reward while adhering to specific constraints and to immerse the system into this state as quickly as possible. We measure the principle's performance in terms of regret and constraint violation. In cases where the action set is finite, we propose an algorithm Optimistic-Pessimistic Convergence and Confidence Bounds (OP-C2B) that ensures sublinear regret and constraint violation simultaneously. Particularly, OP-C2B achieves logarithmic regret and constraint violation when the system convergence rate is linear or superlinear. Furthermore, we generalize our algorithm OP-C2B to the case of an infinite action set and demonstrate its ability to maintain sublinear regret and constraint violation.

Learning the Optimal Control for Evolving Systems with Converging Dynamics

We consider a principle or controller that can pick actions from a fixed action set to control an evolving system with converging dynamics. The actions are interpreted as different configurations or policies. We consider systems with converging dynamics, i.e., if the principle holds the same action, the system will asymptotically converge (possibly requiring a significant amount of time) to a unique stable state determined by this action. This phenomenon can be observed in diverse domains such as epidemic control, computing systems, and markets. In our model, the dynamics of the system are unknown to the principle, and the principle can only receive bandit feedback (maybe noisy) on the impacts of his actions. The principle aims to learn which stable state yields the highest reward while adhering to specific constraints (i.e., optimal stable state) and to immerse the system into this state as quickly as possible. A unique challenge in our model is that the principle has no prior knowledge about the stable state of each action, but waits for the system to converge to the suboptimal stable states costs valuable time. We measure the principle's performance in terms of regret and constraint violation. In cases where the action set is finite, we propose a novel algorithm, termed Optimistic-Pessimistic Convergence and Confidence Bounds (OP-C2B), that knows to switch an action quickly if it is not worth waiting until the stable state is reached. This is enabled by employing "convergence bounds" to determine how far the system is from the stable states, and choosing actions through maintaining a pessimistic assessment of the set of feasible actions while acting optimistically within this set. We establish that OP-C2B can ensure sublinear regret and constraint violation simultaneously. Particularly, OP-C2B achieves logarithmic regret and constraint violation when the system convergence rate is linear or superlinear. Furthermore, we generalize our algorithm OP-C2B to the case of an infinite action set and demonstrate its ability to maintain sublinear regret and constraint violation. We finally show two game control problems including mobile crowdsensing and resource allocation that our model can address.

Push-sum Distributed Dual Averaging Online Convex Optimization With Bandit Feedback

Push-sum Distributed Dual Averaging Online Convex Optimization With Bandit Feedback

Toward joint utilization of absolute and relative bandit feedback for conversational recommendation

Toward joint utilization of absolute and relative bandit feedback for conversational recommendation

The Online Saddle Point Problem and Online Convex Optimization with Knapsacks

We study the online saddle point problem, an online learning problem where at each iteration, a pair of actions needs to be chosen without knowledge of the current and future (convex-concave) payoff functions. The objective is to minimize the gap between the cumulative payoffs and the saddle point value of the aggregate payoff function, which we measure using a metric called saddle point regret (SP-Regret). The problem generalizes the online convex optimization framework, but here, we must ensure that both players incur cumulative payoffs close to that of the Nash equilibrium of the sum of the games. We propose an algorithm that achieves SP-Regret proportional to [Formula: see text] in the general case, and [Formula: see text] SP-Regret for the strongly convex-concave case. We also consider the special case where the payoff functions are bilinear and the decision sets are the probability simplex. In this setting, we are able to design algorithms that reduce the bounds on SP-Regret from a linear dependence in the dimension of the problem to a logarithmic one. We also study the problem under bandit feedback and provide an algorithm that achieves sublinear SP-Regret. We then consider an online convex optimization with knapsacks problem motivated by a wide variety of applications, such as dynamic pricing, auctions, and crowdsourcing. We relate this problem to the online saddle point problem and establish [Formula: see text] regret using a primal-dual algorithm.

Distributed Online Stochastic-Constrained Convex Optimization With Bandit Feedback.

This article studies the distributed online stochastic convex optimization problem with the time-varying constraint over a multiagent system constructed by various agents. The sequences of cost functions and constraint functions, both of which have dynamic parameters following time-varying distributions, are unacquainted to the agent ahead of time. Agents in the network are able to interact with their neighbors through a sequence of strongly connected and time-varying graphs. We develop the adaptive distributed bandit primal-dual algorithm whose step size and regularization sequences are adaptive and have no prior knowledge about the total iteration span T . The adaptive distributed bandit primal-dual algorithm applies bandit feedback with a one-point or two-point gradient estimator to evaluate gradient values. It is illustrated in this article that if the drift of the benchmark sequence is sublinear, then the adaptive distributed bandit primal-dual algorithm exhibits sublinear expected dynamic regret and constraint violation using both two kinds of gradient estimator to compute gradient information. We present a numerical experiment to show the performance of the proposed method.

Safe Pricing Mechanisms for Distributed Resource Allocation with Bandit Feedback

Safe Pricing Mechanisms for Distributed Resource Allocation with Bandit Feedback

Efficient UCB-Based Assignment Algorithm Under Unknown Utility with Application in Mentor-Mentee Matching

The assignment problem, crucial in various real-world applications, involves optimizing the allocation of agents to tasks for maximum utility. While it has been well-studied in the optimization literature when the underlying utilities between all agent-task pairs are known, research is sparse when the utilities are unknown and need to be learned from data on the fly. This paper addresses this gap, as motivated by mentor-mentee matching programs at many U.S. universities. We develop an efficient sequential assignment algorithm, with the aim of nearly maximizing the overall utility simultaneously over different time periods. Our proposed algorithm is to use stochastic bandit feedback to adaptively estimate the unknown utilities through linear regression models, integrating the Upper Confidence Bound (UCB) algorithm in the multi-armed bandit problem with the Hungarian algorithm in the assignment problem. We provide theoretical bounds of our algorithm for both the estimation error and the total regret. Additionally, numerical studies are also conducted to demonstrate the practical effectiveness of our algorithm.

Master-Slave Deep Architecture for Top- K Multiarmed Bandits With Nonlinear Bandit Feedback and Diversity Constraints.

We propose a novel master-slave architecture to solve the top- K combinatorial multiarmed bandits (CMABs) problem with nonlinear bandit feedback and diversity constraints, which, to the best of our knowledge, is the first combinatorial bandits setting considering diversity constraints under bandit feedback. Specifically, to efficiently explore the combinatorial and constrained action space, we introduce six slave models with distinguished merits to generate diversified samples well balancing rewards and constraints as well as efficiency. Moreover, we propose teacher learning-based optimization and the policy cotraining technique to boost the performance of the multiple slave models. The master model then collects the elite samples provided by the slave models and selects the best sample estimated by a neural contextual UCB-based network (NeuralUCB) to decide on a tradeoff between exploration and exploitation. Thanks to the elaborate design of slave models, the cotraining mechanism among slave models, and the novel interactions between the master and slave models, our approach significantly surpasses existing state-of-the-art algorithms in both synthetic and real datasets for recommendation tasks. The code is available at https://github.com/huanghanchi/Master-slave-Algorithm-for-Top-K-Bandits.

Distributed No-regret Learning in Aggregative Games with Residual Bandit Feedback

Distributed No-regret Learning in Aggregative Games with Residual Bandit Feedback

No-regret learning for repeated non-cooperative games with lossy bandits

This paper considers no-regret learning for repeated continuous-kernel games with lossy bandit feedback. Since it is difficult to give an explicit model of the utility functions in dynamic environments, the players’ actions can only be learned with bandit feedback. Moreover, due to unreliable communication channels or privacy protection, the bandit feedback may be lost or dropped at random. Therefore, we study the asynchronous online learning strategy of the players to adaptively adjust the next actions for minimizing the long-term regret loss. The paper provides a novel no-regret learning algorithm, called Online Gradient Descent with lossy bandits (OGD-lb). We first give the regret analysis for concave games with differentiable and Lipschitz utilities. Then we show that the action profile converges to a Nash equilibrium with probability 1 when the game is also strictly monotone. We further provide the mean-squared convergence rate ONpi−2k−1/3 when the game is β-strongly monotone, where N denotes the number of players and pi is the update probability. In addition, we extend the algorithm to the case when the loss probability of the bandit feedback is unknown, and prove its almost sure convergence to Nash equilibrium for strictly monotone games. Finally, we take the resource management in fog computing as an application example, and carry out numerical experiments to empirically demonstrate the algorithm performance.

Online Learning With Incremental Feature Space and Bandit Feedback

Online learning is a fundamental paradigm for learning from continuous data stream. Tradition online learning approaches usually assume that the feature space of data stream is fixed and the incoming instance can always get the true label after making its prediction. However, in many real-world applications, such as the personalized recommender systems, the feature space may keep expanding due to the accumulation of user behaviors. Besides, we may only get bandit feedback, i.e., we only know whether the prediction is correct or not. To solve this important but rarely studied problem, we propose a novel algorithm LIFBF, together with its two variants LIFBF-I and LIFBF-II, to learn from data stream with incremental feature space and bandit feedback. Specifically, when an instance arrives with augmented features, we first utilize the exploration-exploitation strategy to guess its best label, then, a new loss function considering both bandit feedback and guessed label is proposed. Finally, we design a highly dynamic multi-class classifier, which updates the shared and augmented features by adopting the passive-aggressive rule and structural risk minimization principle, respectively. We theoretically analyze the cumulative loss bound of LIFBF. Besides, empirical studies on various datasets further validate the effectiveness of our proposed algorithms.

Decentralized multi-task stochastic optimization with compressed communications

We consider a multi-agent network where each node has a stochastic (local) cost function that depends on the decision variable of that node and a random variable, and further, the decision variables of neighboring nodes are pairwise constrained. There is an aggregated objective function for the network, composed additively of the expected values of the local cost functions at the nodes, and the overall goal of the network is to obtain the minimizing solution to this aggregate objective function subject to all the pairwise constraints. This is to be achieved at the level of the nodes using decentralized information and local computation, with exchanges of only compressed information allowed by neighboring nodes. The paper develops algorithms and obtains performance bounds for two different models of local information availability at the nodes: (i) sample feedback, where each node has direct access to samples of the local random variable to evaluate its local cost, and (ii) bandit feedback, where samples of the random variables are not available, but only the values of the local cost functions at two random points close to the decision are available to each node. For both models, with compressed communication between neighbors, we have developed decentralized saddle-point algorithms that deliver performances no different (in order sense) from those without communication compression; specifically, we show that deviation from the global minimum value and violations of the constraints are upper-bounded by O(T−12) and O(T−14), respectively, where T is the number of iterations. Numerical examples provided in the paper corroborate these bounds.

Distributed bandit online optimisation for energy management in smart grids

ABSTRACTThis paper presents a distributed optimisation algorithm based on one-point bandit feedback (OPBF) which enables the solving of energy management problems (EMPs) over directed networks. Unlike existing EMPs with known cost functions, the proposed online energy management approach considers a time-varying and unknown cost function, which creates sampling difficulty. To tackle this challenge, a random gradient-free oracle is constructed, allowing for the facilitation of output generation updates. This construction significantly mitigates the need for explicit expressions of the cost function. Furthermore, the proposed algorithm successfully enforces both the supply-demand balance constraint and the generation constraint in EMPs. In order to evaluate performance, this study introduces a performance index referred to as regret, which exhibits sublinear convergence. This finding provides additional evidence that the algorithm can achieve optimal output generation at a rapid convergence rate, subject to certain step-size conditions. Finally, the performance of the algorithm is verified on both a modified 6-bus system and an IEEE 162-bus system. The results demonstrate the effectiveness and efficiency of the proposed algorithm in solving EMPs over directed networks.

Distributed online bandit linear regressions with differential privacy

This paper addresses the distributed online bandit linear regression problems with privacy protection, in which the training data are spread in a multi-agent network. Each node identifies a linear predictor to fit the training data and experiences a square loss on each round. The purpose is to minimize the regret that assesses the difference of the accumulated loss between the online linear predictor and the optimal offline linear predictor. Moreover, the differential privacy strategy is adopted to prevent the adversary from inferring the parameter vector of any node. Two efficient differentially private distributed online regression algorithms are developed in the cases of one-point and two-point bandit feedback. Our analysis suggests that the developed algorithms achieve ϵ-differential privacy and establish the regret upper bounds in O(K3/4) and O(K) for one-point and two-point bandit feedback, respectively, where K is the time horizon. We also show that there exists a tradeoff between our algorithms’ privacy level and convergence. Finally, the performance of the proposed algorithms is validated by a numerical example.

Distributed Task Management in Fog Computing: A Socially Concave Bandit Game

Fog computing leverages the task offloading capabilities at the network’s edge to improve efficiency and enable swift responses to application demands. However, the design of task allocation strategies in a fog computing network is still challenging because of the heterogeneity of fog nodes and uncertainties in system dynamics. We formulate the distributed task allocation problem as a social-concave game with bandit feedback and show that the game has a unique Nash equilibrium, which is implementable using no-regret learning strategies (regret with sublinear growth). We then develop two no-regret online decision-making strategies. One strategy, namely bandit gradient ascent with momentum, is an online convex optimization algorithm with bandit feedback. The other strategy, Lipschitz bandit with initialization, is an EXP3 multi-armed bandit algorithm. We establish regret bounds for both strategies and analyze their convergence characteristics. Moreover, we compare the proposed strategies with an allocation strategy named learning with linear rewards. Theoretical-and numerical analysis shows the superior performance of the proposed strategies for efficient task allocation compared to the state-of-the-art methods.

Online distributed detection of sensor networks with delayed information

Distributed decision on sensor network is affected by the bandwidth, network congestion, and computation capability of sensors, which cause the communication delays and feedback delays during the signal processing process. In this paper, the effects of state constraints, directed graph, communication delays and feedback delays on the online distributed parameter estimation problem for multi-sensor networks are considered comprehensively. To this end, we first propose an Online Decentralized Gradient Descent Algorithm (ODGDA) to overcome the effects of these factors and give the regret upper bounds for convex loss function and strongly convex loss function, respectively. Further, considering that the loss function may not be fully disclosed in the complex environment, we extend ODGDA to the scenario of bandit feedback and propose an Online Decentralized Bandit Feedback Algorithm (ODBFA), which is updated using historical state information stored by sensors. The analysis shows that, compared with the ODGDA which can directly use the gradient information for state update, the ODBFA utilizing two-point bandit feedback information does not lead to regret degradation in the order sense. Finally, the effectiveness of the proposed algorithm is verified by the multi-sensor network.

Constrained distributed online convex optimization with bandit feedback for unbalanced digraphs

AbstractIn this study, a distributed primal‐dual bandit feedback method for online convex optimization with time‐varying coupled inequality constraints on unbalanced directed graphs is proposed. A multiagent network is considered in which agents exchange the estimations of the dual optimizer and the scaling variable with their neighbors. The scaling variable is used to resolve the bias of the estimations caused by a directed communication network. Each agent does not have prior knowledge of the loss function, and its value at a queried point is sequentially disclosed to each agent. Each agent performs a projected subgradient‐based primal‐dual algorithm to estimate the optimal solution. It is confirmed that both the expected dynamic regret of the loss function and the cumulative error of the constraint violation achieve sublinearity using the proposed online algorithm with the two‐point bandit feedback.

Technical Note—On Adaptivity in Nonstationary Stochastic Optimization with Bandit Feedback

Optimal Nonstationary Optimization Without Knowing Function Changes Nonstationary stochastic optimization plays a vital role in a number of computer science and operations research applications. It is known how to design and analyze algorithms that optimize a sequence of strongly convex/concave and smooth functions with access to only one-point noisy function values with the underlying function sequence subject to maximum magnitude of function changes. In recent work from Wang titled “Technical Note: On Adaptivity in Nonstationary Stochastic Optimization with Bandit Feedback,” an optimization algorithm is designed and analyzed without assuming the magnitude of function changes is known in advance. Optimality of the designed algorithm is demonstrated.

Online bandit convex optimisation with stochastic constraints via two-point feedback

ABSTRACT In this paper, an online convex optimisation problem with stochastic constraints in the bandit setup is investigated. We are particularly interested in the scenario where the gradient information of both loss and constraint functions is unavailable. Under this scenario, only the values of loss and constraint functions at a few random points near the decision are provided to the decision maker after the decision is submitted. We first propose an online bandit algorithm based on the virtual queue in which two-point feedback is used to approximate the gradient feedback. Then we adopt the static benchmark to analyse the optimisation performance and establish the sub-linear expected static regret and sub-linear expected constraint violations of the proposed algorithm in the two-point bandit feedback setup. Moreover, the expected static regret and constraint violations are further improved to when loss functions satisfy the condition of strong convexity. Finally, an online job scheduling numerical simulation is shown to demonstrate the performance of the proposed method and to corroborate the theoretical guarantees.