Average Regret Research Articles

Distributed inertial online game algorithm for tracking generalized Nash equilibria.

This paper is concerned with the distributed generalized Nash equilibrium (GNE) tracking problem of noncooperative games in dynamic environments, where the cost function and/or the coupled constraint function are time-varying and revealed to each agent after it makes a decision. We first consider the case without coupled constraints and propose a distributed inertial online game (D-IOG) algorithm based on the mirror descent method. The proposed algorithm is capable of tracking Nash equilibrium (NE) through a time-varying communication graph and has the potential of achieving a low average regret. With an appropriate non-increasing stepsize sequence and an inertial parameter, the regrets can grow sublinearly if the deviation of the NE sequence grows sublinearly. Second, the time-varying coupled constraints are further investigated, and a modified D-IOG algorithm for tracking GNE is proposed based on the primal-dual and mirror descent methods. Then, the upper bounds of regrets and constraint violation are derived. Moreover, inertia and two information transmission modes are discussed. Finally, two simulation examples are provided to illustrate the effectiveness of the D-IOG algorithms.

Fairness and Welfare Quantification for Regret in Multi-Armed Bandits

We extend the notion of regret with a welfarist perspective. Focussing on the classic multi-armed bandit (MAB) framework, the current work quantifies the performance of bandit algorithms by applying a fundamental welfare function, namely the Nash social welfare (NSW) function. This corresponds to equating algorithm's performance to the geometric mean of its expected rewards and leads us to the study of Nash regret, defined as the difference between the - a priori unknown - optimal mean (among the arms) and the algorithm's performance. Since NSW is known to satisfy fairness axioms, our approach complements the utilitarian considerations of average (cumulative) regret, wherein the algorithm is evaluated via the arithmetic mean of its expected rewards. This work develops an algorithm that, given the horizon of play T, achieves a Nash regret of O ( sqrt{(k log T)/T} ), here k denotes the number of arms in the MAB instance. Since, for any algorithm, the Nash regret is at least as much as its average regret (the AM-GM inequality), the known lower bound on average regret holds for Nash regret as well. Therefore, our Nash regret guarantee is essentially tight. In addition, we develop an anytime algorithm with a Nash regret guarantee of O( sqrt{(k log T)/T} log T ).

OnAI-Comp: An Online AI Experts Competing Framework for Early Sepsis Detection.

Sepsis is a major public concern due to its high mortality, morbidity, and financial cost. There are many existing works of early sepsis prediction using different machine learning models to mitigate the outcomes brought by sepsis. In the practical scenario, the dataset grows dynamically as new patients visit the hospital. Most existing models, being "offline" models and having used retrospective observational data, cannot be updated and improved dynamically using the new observational data. Incorporating the new data to improve the offline models requires retraining the model, which is very computationally expensive. To solve the challenge mentioned above, we propose an Online Artificial Intelligence Experts Competing Framework (OnAI-Comp) for early sepsis detection using an online learning algorithm called Multi-armed Bandit. We selected several machine learning models as the artificial intelligence experts and used average regret to evaluate the performance of our model. The experimental analysis demonstrated that our model would converge to the optimal strategy in the long run. Meanwhile, our model can provide clinically interpretable predictions using existing local interpretable model-agnostic explanation technologies, which can aid clinicians in making decisions and might improve the probability of survival.

Sparse assortment personalization in high dimensions

The data-driven conditional multinomial logit choice model with customer features performs well in the assortment personalization problem when the low-rank structure of the parameter matrix is considered. However, despite recent theoretical and algorithmic advances, parameter estimation in the choice model still poses a challenging task, especially when there are more predictors than observations. For this reason, we suggest a penalized likelihood approach based on a feature matrix to recover the sparse structure from populations and products toward the assortment. Our proposed method considers simultaneously low-rank and sparsity structures, which can further reduce model complexity and improve its estimation and prediction accuracy. A new algorithm, sparse factorial gradient descent (SFGD), was proposed to estimate the parameter matrix, which has high interpretability and efficient computing performance. As a first-order method, the SFGD works well in high-dimensional scenarios because of the absence of the Hessian matrix. Simulation studies show that the SFGD algorithm outperforms state-of-the-art methods in terms of estimation, sparsity recovery, and average regret. We also demonstrate the effectiveness of our proposed method using advertising behavior data analysis.

Algorithms for Average Regret Minimization

In this paper, we study a problem from the realm of multi-criteria decision making in which the goal is to select from a given set S of d-dimensional objects a minimum sized subset S' with bounded regret. Thereby, regret measures the unhappiness of users which would like to select their favorite object from set S but now can only select their favorite object from the subset S'. Previous work focused on bounding the maximum regret which is determined by the most unhappy user. We propose to consider the average regret instead which is determined by the sum of (un)happiness of all possible users. We show that this regret measure comes with desirable properties as supermodularity which allows to construct approximation algorithms. Furthermore, we introduce the regret minimizing permutation problem and discuss extensions of our algorithms to the recently proposed k-regret measure. Our theoretical results are accompanied with experiments on a variety of inputs with d up to 7.

Nested-Solution Facility Location Models.

Classical facility location models can generate solutions that do not maintain consistency in the set of utilized facilities as the number of utilized facilities is varied. We introduce the concept of nested facility locations, in which the solution utilizing p facilities is a subset of the solution utilizing q facilities, for all i ≤ p < q ≤ j, given some lower limit i and upper limit j on r, the number of facilities that will be utilized in the future. This approach is demonstrated with application to the p-median model, with computational testing showing these new models achieve reductions in both average regret and worst-case regret when r ≠ p facilities are actually utilized.

Sample Efficient Reinforcement Learning with REINFORCE

Policy gradient methods are among the most effective methods for large-scale reinforcement learning, and their empirical success has prompted several works that develop the foundation of their global convergence theory. However, prior works have either required exact gradients or state-action visitation measure based mini-batch stochastic gradients with a diverging batch size, which limit their applicability in practical scenarios. In this paper, we consider classical policy gradient methods that compute an approximate gradient with a single trajectory or a fixed size mini-batch of trajectories under soft-max parametrization and log-barrier regularization, along with the widely-used REINFORCE gradient estimation procedure. By controlling the number of "bad" episodes and resorting to the classical doubling trick, we establish an anytime sub-linear high probability regret bound as well as almost sure global convergence of the average regret with an asymptotically sub-linear rate. These provide the first set of global convergence and sample efficiency results for the well-known REINFORCE algorithm and contribute to a better understanding of its performance in practice.

Data-driven Clustering and Feature-based Retail Electricity Pricing with Smart Meters

We consider an electric utility company that serves retail electricity customers over a discrete-time horizon. In each period, the company observes the customers' consumption as well as high-dimensional features on customer characteristics and exogenous factors. These features exhibit three types of heterogeneity—over time, customers, or both. Based on the consumption and feature observations, the company can dynamically adjust the retail electricity price at the customer level. The consumption depends on the features: there is an underlying structure of clusters in the feature space, and the relationship between consumption and features is different in each cluster. Initially, the company knows neither the underlying cluster structure nor the corresponding consumption models. We measure the company's performance by its average regret, i.e., the profit loss per period per customer, relative to a clairvoyant who knows the underlying cluster structure and the consumption model in each cluster. We design a data-driven policy of joint spectral clustering and feature-based pricing, and show that its average regret converges to zero at the fastest achievable rate. We conduct case studies based on real-life smart meter data from Texas and simulation experiments. Relative to the company policy, our policy increases company profits by 146% over a three-month period.

Online reinforcement learning for adaptive interference coordination

AbstractHeterogeneous networks (HetNets), in which small cells overlay macro cells, are a cost‐effective approach to increase the capacity of cellular networks. However, HetNets have raised new issues related to cell association and interference management. In particular, the optimal configuration of interference coordination (IC) parameters is a challenging task because it depends on multiple stochastic processes such as the locations of the users, the traffic demands, or the strength of the received signals. This work proposes a self‐optimization algorithm capable of finding the optimal configuration in an operating network. We address the problem using a reinforcement learning (RL) approach, in which the actions are the IC configurations, whose performances are initially unknown. The main difficulty is that, due to the variable network conditions, the performance of each action may change over time. Our proposal is based on two main elements: the sequential exploration of subsets of actions (exploration regions), and an optimal stopping (OS) strategy for deciding when to end current exploration and start a new one. For our algorithm, referred to as local exploration with optimal stopping (LEOS), we provide theoretical bounds on its long‐term regret per sample and its convergence time. We compare LEOS to state‐of‐the‐art learning algorithms based on multiarmed bandits and policy gradient RL. Considering different changing rates in the network conditions, our numerical results show that LEOS outperforms the first alternative by 22%, and the second one by 48% in terms of average regret per sample.

An Approximation Approach for Response-Adaptive Clinical Trial Design

Multiarmed bandit (MAB) problems, typically modeled as Markov decision processes (MDPs), exemplify the learning versus earning trade-off. An area that has motivated theoretical research in MAB designs is the study of clinical trials, where the application of such designs has the potential to significantly improve patient outcomes. However, for many practical problems of interest, the state space is intractably large, rendering exact approaches to solving MDPs impractical. In particular, settings that require multiple simultaneous allocations lead to an expanded state and action-outcome space, necessitating the use of approximation approaches. We propose a novel approximation approach that combines the strengths of multiple methods: grid-based state discretization, value function approximation methods, and techniques for a computationally efficient implementation. The hallmark of our approach is the accurate approximation of the value function that combines linear interpolation with bounds on interpolated value and the addition of a learning component to the objective function. Computational analysis on relevant datasets shows that our approach outperforms existing heuristics (e.g., greedy and upper confidence bound family of algorithms) and a popular Lagrangian-based approximation method, where we find that the average regret improves by up to 58.3%. A retrospective implementation on a recently conducted phase 3 clinical trial shows that our design could have reduced the number of failures by 17% relative to the randomized control design used in that trial. Our proposed approach makes it practically feasible for trial administrators and regulators to implement Bayesian response-adaptive designs on large clinical trials with potential significant gains.

Expansion Planning Method of the Industrial Park Integrated Energy System Considering Regret Aversion

Industrial parks have various sources and conversion forms of energy. The many uncertainties in the planning of industrial park integrated energy systems (IPIES) pose a great risk of regret in planning schemes; thus, an expansion planning method for an IPIES, considering regret aversion, is proposed. Based on comprehensive regret value consisting of min–max regret aversion and the min average regret value, the method optimizes the comprehensive cost of the expansion planning scheme in IPIES under different natural gas price fluctuation scenarios, including costs of construction, operation and maintenance, and environmental protection. A multi-stage expansion planning scheme and typical daily operation plans under multiple natural gas price fluctuation scenarios of the IPIES in an economic and technological development zone in southeast China are used to demonstrate the validity of the method. The results show that, compared with a traditional planning method based on expectation, the proposed expansion planning method could reduce the maximum regret value by 14% on average, and greatly reduces the risk of decision-making regret by up to 18%. At the same time, the influence of natural gas price on expansion planning of the IPIES is discussed.

Algorithms for Average Regret Minimization

In this paper, we study a problem from the realm of multicriteria decision making in which the goal is to select from a given set S of d-dimensional objects a minimum sized subset S0 with bounded regret. Thereby, regret measures the unhappiness of users which would like to select their favorite object from set S but now can only select their favorite object from the subset S0. Previous work focused on bounding the maximum regret which is determined by the most unhappy user. We propose to consider the average regret instead which is determined by the sum of (un)happiness of all possible users. We show that this regret measure comes with desirable properties as supermodularity which allows to construct approximation algorithms. Furthermore, we introduce the regret minimizing permutation problem and discuss extensions of our algorithms to the recently proposed k-regret measure. Our theoretical results are accompanied with experiments on a variety of inputs with d up to 7.

Distributed Capacity Allocation of Shared Energy Storage Using Online Convex Optimization

This paper studies capacity allocation of an energy storage (ES) device which is shared by multiple homes in smart grid. Given a time-of-use (TOU) tariff, homes use the ES to shift loads from peak periods to off-peak periods, reducing electricity bills. In the proposed ES sharing model, the ES capacity has to be allocated to homes before the homes’ load data is completely known. To this end, an online ES capacity allocation algorithm is developed based on the online convex optimization framework. Under the online algorithm, the complex allocation problem can be solved round by round: at each round, the algorithm observes current system states and predicts a decision for the next round. The proposed algorithm is able to minimize homes’ costs by learning from home load data in a serial fashion. It is proven that the online algorithm can ensure zero average regret and long-term budget balance of homes. Further, a distributed implementation of the online algorithm is proposed based on alternating direction method of multipliers framework. In the distributed implementation, the one-round system problem is decomposed into multiple subproblems that can be solved by homes locally, so that an individual home does not need to send its private load data to any other. In simulation, actual home load data and a TOU tariff of the United States are used. Results show that the proposed online approach leads to the lowest home costs, compared to other benchmark approaches.

Individual Regret Bounds for the Distributed Online Alternating Direction Method of Multipliers

We consider a distributed online optimization problem where, at each time, a group of agents choose their individual states, after which an individual cost function is revealed to each of them. The whole network then faces a regret according to the cumulative sum of costs incurred by the agents’ chosen states, and each agent faces an individual regret according to the cumulative sum of costs incurred by the agent's state estimation, perceived as the whole network's chosen state. In order to tackle the minimization of the individual regret using only local information, we assume that the group of agents communicate over a fixed undirected connected graph. We then propose an online version of the alternating direction method of multipliers algorithm, distributed over the communication graph, which allows each agent to drive its individual average regret over time to zero.

An Approximation Approach for Response Adaptive Clinical Trial Design

Multi-armed bandit (MAB) problems, typically modeled as Markov decision processes (MDPs), exemplify the learning vs. earning tradeoff. An area that has motivated theoretical research in MAB designs is the study of clinical trials, where the application of such designs has the potential to significantly improve patient outcomes. However, for many practical problems of interest, the state space is intractably large, rendering exact approaches to solving MDPs impractical. In particular, settings that require multiple simultaneous allocations lead to an expanded state and action-outcome space, necessitating the use of approximation approaches. We propose a novel approximation approach that combines the strengths of multiple methods: grid-based state discretization, value function approximation methods, and techniques for a computationally efficient implementation. The hallmark of our approach is the accurate approximation of the value function that combines linear interpolation with bounds on interpolated value and the addition of a learning component to the objective function. Computational analysis on relevant datasets shows that our approach outperforms existing heuristics (e.g. greedy and upper confidence bound family of algorithms) as well as a popular Lagrangian-based approximation method, where we find that the average regret improves by up to 58.3%. A retrospective implementation on a recently conducted phase 3 clinical trial shows that our design could have reduced the number of failures by 17% relative to the randomized control design used in that trial. Our proposed approach makes it practically feasible for trial administrators and regulators to implement Bayesian response-adaptive designs on large clinical trials with potential significant gains.

Coactive Critiquing: Elicitation of Preferences and Features

When faced with complex choices, users refine their own preference criteria as they explore the catalogue of options. In this paper we propose an approach to preference elicitation suited for this scenario. We extend Coactive Learning, which iteratively collects manipulative feedback, to optionally query example critiques. User critiques are integrated into the learning model by dynamically extending the feature space. Our formulation natively supports constructive learning tasks, where the option catalogue is generated on-the-fly. We present an upper bound on the average regret suffered bythe learner. Our empirical analysis highlights the promise of our approach.

Pollution Source Identification in Groundwater Systems: Application of Regret Theory and Bayesian Networks

Pollution source identification in groundwater resources is a challenging task due to existing uncertainties in both pollutant source characteristics and mass transport in porous media. To obtain a robust and cost-effective groundwater monitoring configuration, this paper presents a new regret-based optimization model which minimizes the number of monitoring wells and average regret in estimating undetected polluted area. A Monte Carlo analysis is used to consider existing uncertainties in both pollution source characteristics and parameters of groundwater quality simulation model. MODFLOW and MT3D, groundwater quantity and quality simulation models, are used to simulate the spatial and temporal variations of a water quality indicator in groundwater. For each non-dominated solution provided by a bi-objective optimization model, the optimal positions of monitoring wells are also determined by minimizing the corresponding regret in undetected polluted area. Furthermore, a Bayesian network (BN) is trained and validated based on results of the Monte Carlo analysis. The trained BN is capable of accurately determining an unknown pollution source using monitoring data in real-time operation of monitoring system. To demonstrate the efficiency and applicability of the proposed methodology, it is applied to the Tehran Aquifer in the Tehran refinery region, which is highly polluted due to leakage from several tanks of petroleum products.

Near-optimal no-regret algorithms for zero-sum games

We propose a new no-regret learning algorithm. When used against an adversary, our algorithm achieves average regret that scales optimally as O(1T) with the number T of rounds. However, when our algorithm is used by both players of a zero-sum game, their average regret scales as O(ln⁡TT), guaranteeing a near-linear rate of convergence to the value of the game. This represents an almost-quadratic improvement on the rate of convergence to the value of a zero-sum game known to be achievable by any no-regret learning algorithm. Moreover, it is essentially optimal as we also show a lower bound of Ω(1T) for all distributed dynamics, as long as the players do not know their payoff matrices in the beginning of the dynamics. (If they did, they could privately compute minimax strategies and play them ad infinitum.)

Parallelizing exploration-exploitation tradeoffs in Gaussian process bandit optimization

How can we take advantage of opportunities for experimental parallelization in exploration-exploitation tradeoffs? In many experimental scenarios, it is often desirable to execute experiments simultaneously or in batches, rather than only performing one at a time. Additionally, observations may be both noisy and expensive. We introduce Gaussian Process Batch Upper Confidence Bound (GP-BUCB), an upper confidence bound-based algorithm, which models the reward function as a sample from a Gaussian process and which can select batches of experiments to run in parallel. We prove a general regret bound for GP-BUCB, as well as the surprising result that for some common kernels, the asymptotic average regret can be made independent of the batch size. The GP-BUCB algorithm is also applicable in the related case of a delay between initiation of an experiment and observation of its results, for which the same regret bounds hold. We also introduce Gaussian Process Adaptive Upper Confidence Bound (GP-AUCB), a variant of GP-BUCB which can exploit parallelism in an adaptive manner. We evaluate GP-BUCB and GP-AUCB on several simulated and real data sets. These experiments show that GP-BUCB and GP-AUCB are competitive with state-of-the-art heuristics.

Efficient online learning for multitask feature selection

Learning explanatory features across multiple related tasks, or MultiTask Feature Selection (MTFS), is an important problem in the applications of data mining, machine learning, and bioinformatics. Previous MTFS methods fulfill this task by batch-mode training. This makes them inefficient when data come sequentially or when the number of training data is so large that they cannot be loaded into the memory simultaneously. In order to tackle these problems, we propose a novel online learning framework to solve the MTFS problem. A main advantage of the online algorithm is its efficiency in both time complexity and memory cost. The weights of the MTFS models at each iteration can be updated by closed-form solutions based on the average of previous subgradients. This yields the worst-case bounds of the time complexity and memory cost at each iteration, both in the order of O ( d × Q ), where d is the number of feature dimensions and Q is the number of tasks. Moreover, we provide theoretical analysis for the average regret of the online learning algorithms, which also guarantees the convergence rate of the algorithms. Finally, we conduct detailed experiments to show the characteristics and merits of the online learning algorithms in solving several MTFS problems.