Total Regret Research Articles

Multi-Armed Bandit-Based User Network Node Selection.

In the scenario of an integrated space-air-ground emergency communication network, users encounter the challenge of rapidly identifying the optimal network node amidst the uncertainty and stochastic fluctuations of network states. This study introduces a Multi-Armed Bandit (MAB) model and proposes an optimization algorithm leveraging dynamic variance sampling (DVS). The algorithm posits that the prior distribution of each node's network state conforms to a normal distribution, and by constructing the distribution's expected value and variance, it maximizes the utilization of sample data, thereby maintaining an equilibrium between data exploitation and the exploration of the unknown. Theoretical substantiation is provided to illustrate that the Bayesian regret associated with the algorithm exhibits sublinear growth. Empirical simulations corroborate that the algorithm in question outperforms traditional ε-greedy, Upper Confidence Bound (UCB), and Thompson sampling algorithms in terms of higher cumulative rewards, diminished total regret, accelerated convergence rates, and enhanced system throughput.

Efficient UCB-Based Assignment Algorithm Under Unknown Utility with Application in Mentor-Mentee Matching

The assignment problem, crucial in various real-world applications, involves optimizing the allocation of agents to tasks for maximum utility. While it has been well-studied in the optimization literature when the underlying utilities between all agent-task pairs are known, research is sparse when the utilities are unknown and need to be learned from data on the fly. This paper addresses this gap, as motivated by mentor-mentee matching programs at many U.S. universities. We develop an efficient sequential assignment algorithm, with the aim of nearly maximizing the overall utility simultaneously over different time periods. Our proposed algorithm is to use stochastic bandit feedback to adaptively estimate the unknown utilities through linear regression models, integrating the Upper Confidence Bound (UCB) algorithm in the multi-armed bandit problem with the Hungarian algorithm in the assignment problem. We provide theoretical bounds of our algorithm for both the estimation error and the total regret. Additionally, numerical studies are also conducted to demonstrate the practical effectiveness of our algorithm.

Multi-period consequence management of contaminations in water distribution networks: application of regret-based optimization model

ABSTRACT This study presents a Regret-based optimization model for multi-period consequence management of contaminations entering the WDNs considering the pollutants loads’ entry uncertainties. For this purpose, first, the EPANET software is utilized as a water quality-quantity simulation modeling tool. Then, an optimization model based on a genetic algorithm using the primary objective functions of minimizing the return time of the contaminated WDNs to the normal mode, the amount of pollution, and the number of polluted nodes. Besides, the pollutant load-related uncertainties are included in the presented simulation-optimization model by minimizing the maximum Regret (MMR) and the total Regret (MTR) in two steady and dynamic status for all three mentioned objective functions. Using these nine objective functions, three management instruments are utilized: quick closing valves, discharging hydrants, and boosting pumps. The results show that the proposed model can effectively prevent harmful pollution by offering effective management strategies, mainly in dynamic status scenarios.

Shrinking the Upper Confidence Bound: A Dynamic Product Selection Problem for Urban Warehouses

The recent rising popularity of ultrafast delivery services on retail platforms fuels the increasing use of urban warehouses, whose proximity to customers makes fast deliveries viable. The space limit in urban warehouses poses a problem for such online retailers: the number of stock keeping units (SKUs) they carry is no longer “the more, the better,” yet it can still be significantly large, reaching hundreds or thousands in a product category. In this paper, we study algorithms for dynamically selecting a large number of products (i.e., SKUs) with top customer purchase probabilities on the fly, from an ocean of potential products to offer on retailers’ ultrafast delivery platforms. We distill the product selection problem into a semibandit model with linear generalization. There are in total N arms corresponding to N products, each with a feature vector of dimension d. The player pulls K arms in each period and observes the bandit feedback from each of the pulled arms. We focus on the setting where K is much greater than the number of total time periods T or the dimension of product features d. We first analyze a standard Upper Confidence Bound (UCB) algorithm and show its regret bound can be expressed as the sum of a T-independent part and a T-dependent part, which we refer to as “fixed cost” and “variable cost,” respectively. To reduce the fixed cost for large K values, we propose a novel online learning algorithm, which iteratively shrinks the upper confidence bounds within each period, and show its fixed cost is reduced by a factor of d. Moreover, we test the algorithms on an industrial data set from Alibaba Group. Experimental results show that our new algorithm reduces the total regret of the standard UCB algorithm by at least 10%. This paper was accepted by J. George Shanthikumar, big data analytics.

A Near-Optimal Bidding Strategy for Real-Time Display Advertising Auctions

This article introduces a near-optimal bidding algorithm for use in real-time display advertising auctions. These auctions constitute a dominant distribution channel for internet display advertising and a potential funding model for addressable media. The proposed efficient, implementable learning algorithm is proven to rapidly converge to the optimal strategy while achieving zero regret and constituting a competitive equilibrium. This is the first algorithmic solution to the online knapsack problem to offer such theoretical guarantees without assuming a priori knowledge of object values or costs. Furthermore, it meets advertiser requirements by accommodating any valuation metric while satisfying budget constraints. Across a series of 100 simulated and 10 real-world campaigns, the algorithm delivers 98% of the value achievable with perfect foresight and outperforms the best available alternative by 11%. Finally, we show how the algorithm can be augmented to simultaneously estimate impression values and learn the bidding policy. Across a series of simulations, we show that the total regret delivered under this dual objective is less than that from any competing algorithm required only to learn the bidding policy.

A multi-stage emergency supplies pre-allocation approach for freeway black spots: A Chinese case study.

This study presents a multi-stage random regret minimization (RRM) model as an emergency rescue decision support system to determine the emergency resource pre-allocation schedule for the freeway network. The proposed methodology consists of three steps: (1) improved accident frequency approach to identify the black spots on the freeway network, (2) stochastic programming (SP) model to determine the initial allocation plan sets, and (3) regret-based model in the logarithmical specification to select the most minimal regret one considering the factors of the response time, total cost and demand. The model is applied to the case study of 2014–2016 freeway network in Shandong, China. The results show that the random regret minimization (RRM) model can improve the full-compensation of SP model to a certain degree. RRM in logarithmical specification performs lightly better than random utility maximization (RUM) and RRM in the linear-additive specification in this case. This approach emerges as a valuable tool to help decision makers to allocate resources before traffic accident occurs, with the aim of minimizing the total regret of their decisions.

Distributed algorithm under cooperative or competitive priority users in cognitive networks

Opportunistic spectrum access (OSA) problem in cognitive radio (CR) networks allows a secondary (unlicensed) user (SU) to access a vacant channel allocated to a primary (licensed) user (PU). By finding the availability of the best channel, i.e., the channel that has the highest availability probability, a SU can increase its transmission time and rate. To maximize the transmission opportunities of a SU, various learning algorithms are suggested: Thompson sampling (TS), upper confidence bound (UCB), ε-greedy, etc. In our study, we propose a modified UCB version called AUCB (Arctan-UCB) that can achieve a logarithmic regret similar to TS or UCB while further reducing the total regret, defined as the reward loss resulting from the selection of non-optimal channels. To evaluate AUCB’s performance for the multi-user case, we propose a novel uncooperative policy for a priority access where the kth user should access the kth best channel. This manuscript theoretically establishes the upper bound on the sum regret of AUCB under the single or multi-user cases. The users thus may, after finite time slots, converge to their dedicated channels. It also focuses on the Quality of Service AUCB (QoS-AUCB) using the proposed policy for the priority access. Our simulations corroborate AUCB’s performance compared to TS or UCB.

Multi-criteria spherical fuzzy regret based evaluation of healthcare equipment stocks

The catastrophes due to widespread outbreaks create a long-standing distraction and have an accelerating transmission. The uncontrolled outbreaks cause not only health-related problems but also supply chain related problems. The outbreak caused by the coronavirus (COVID-19) shows how vulnerable the Healthcare systems and the supporting systems such as supply chains of the countries to such type of disasters. Keeping high levels of inventory, especially for healthcare products, can be beneficial to overcome such shortage problems. Nevertheless, keeping a high level of inventory can be costly, and the durability of the products creates a limit. The decision-makers have to carefully decide the inventory levels by considering many factors such as the criticality of the product and the easiness of producing the product. In this study, we try to develop a decision model for defining the inventory levels in Healthcare systems by considering multiple scenarios such as outbreaks. A novel spherical regret based multi-criteria decision-making approach is developed and used for evaluating the total regret of not keeping stock of the healthcare equipment.

Conservative Exploration for Semi-Bandits with Linear Generalization: A Product Selection Problem for Urban Warehouses

The recent rising popularity of ultra-fast delivery services on retail platforms fuels increasing use of urban warehouses, whose proximity to customers makes fast deliveries viable. The space limit in urban warehouses poses a problem for online retailers: number of products (SKUs) they carry is no longer the more, better, yet it can still be significantly large, reaching hundreds or thousands in a product category. In this paper, we study algorithms for dynamically selecting a large number of products (i.e., SKUs) with top customer purchase probabilities on fly, from an ocean of potential products to offer on retailers' ultra-fast delivery platforms. We distill product selection problem into a semi-bandit model with linear generalization. There are in total N different arms, each with a feature vector of dimension d. The player pulls K arms in each period and observes bandit feedback from each of pulled arms. We focus on setting where K is much greater than number of total time periods T or dimension of product features d. We first analyze a standard UCB algorithm and show its regret bound can be expressed as sum of a T-independent part O(Kd3/2) and a T-dependent part O(d √(KT)), which we refer to as and variable respectively. To reduce fixed cost for large K values, we propose a novel online learning algorithm, which iteratively shrinks upper confidence bounds within each period, and show its fixed cost is reduced by a factor of d to O(K √(d)). Moreover, we test algorithms on an industrial dataset from Alibaba Group. Experimental results show that our new algorithm reduces total regret of standard UCB algorithm by at least 10%.

Dynamic Pricing to Minimize Maximum Regret

We consider a dynamic pricing problem that involves selling a given inventory of a single product over a short, two‐period selling season. There is insufficient time to replenish inventory during this season, hence sales are made entirely from inventory. The demand for the product is a stochastic, nonincreasing function of price. We assume interval uncertainty for demand, that is, knowledge of upper and lower bounds but not a probability distribution, with no correlation between the two periods. We minimize the maximum total regret over the two periods that results from the pricing decisions. We consider a dynamic model where the decision maker chooses the price for each period contingent on the remaining inventory at the beginning of the period, and a static model where the decision maker chooses the prices for both periods at the beginning of the first period. Both models can be solved by a polynomial time algorithm that solves systems of linear inequalities. Our computational study demonstrates that the prices generated by both our models are insensitive to errors in estimating the demand intervals. Our dynamic model outperforms our static model and two classical approaches that do not use demand probability distributions, when evaluated by maximum regret, average relative regret, variability, and risk measures. Further, our dynamic model generates a total expected revenue which closely approximates that of a maximum expected revenue approach which requires demand probability distributions.

Stochastic regret minimization for revenue management problems with nonstationary demands

AbstractWe study an admission control model in revenue management with nonstationary and correlated demands over a finite discrete time horizon. The arrival probabilities are updated by current available information, that is, past customer arrivals and some other exogenous information. We develop a regret‐based framework, which measures the difference in revenue between a clairvoyant optimal policy that has access to all realizations of randomness a priori and a given feasible policy which does not have access to this future information. This regret minimization framework better spells out the trade‐offs of each accept/reject decision. We proceed using the lens of approximation algorithms to devise a conceptually simple regret‐parity policy. We show the proposed policy achieves 2‐approximation of the optimal policy in terms of total regret for a two‐class problem, and then extend our results to a multiclass problem with a fairness constraint. Our goal in this article is to make progress toward understanding the marriage between stochastic regret minimization and approximation algorithms in the realm of revenue management and dynamic resource allocation. © 2016 Wiley Periodicals, Inc. Naval Research Logistics 63: 433–448, 2016

SPA: Almost Optimal Accessing of Nonstochastic Channels in Cognitive Radio Networks

In this work, we address the spectrum utilization problem in cognitive radio (CR) networks, in which a CR can only utilize spectrum opportunities when the channel is idle. One challenge for a CR is to balance exploring new channels and exploiting existing channel, due to the fact that the channel availability and channel quality, potentially heterogeneous and time-dependent, are often unknown in advance due to the large number of channels, and the limited hardware capability of single CR. In this work, we propose joint channel sensing, probing, and accessing schemes for secondary users in cognitive radio networks. Our method has time and space complexity $O({N}\cdot {u})$ for a network with ${N}$ channels and ${u}$ secondary users, while applying classic methods requires exponential time complexity. We prove that, even when channel states are selected by adversary (thus nonstochastic), it results in a total regret uniformly upper bounded by $\Theta (\sqrt{T {N}\,\log {N}})$ , w.h.p , for communication lasts for $T$ timeslots. Our protocol can be implemented in a distributed manner due to the nonstochastic channel assumption. Our experiments show that our schemes achieve almost optimal throughput compared with an optimal static strategy, and perform significantly better than previous methods in many settings.

An Integrated Environmental Assessment of Green and Gray Infrastructure Strategies for Robust Decision Making.

The robustness of a range of watershed-scale "green" and "gray" drainage strategies in the future is explored through comprehensive modeling of a fully integrated urban wastewater system case. Four socio-economic future scenarios, defined by parameters affecting the environmental performance of the system, are proposed to account for the uncertain variability of conditions in the year 2050. A regret-based approach is applied to assess the relative performance of strategies in multiple impact categories (environmental, economic, and social) as well as to evaluate their robustness across future scenarios. The concept of regret proves useful in identifying performance trade-offs and recognizing states of the world most critical to decisions. The study highlights the robustness of green strategies (particularly rain gardens, resulting in half the regret of most options) over end-of-pipe gray alternatives (surface water separation or sewer and storage rehabilitation), which may be costly (on average, 25% of the total regret of these options) and tend to focus on sewer flooding and CSO alleviation while compromising on downstream system performance (this accounts for around 50% of their total regret). Trade-offs and scenario regrets observed in the analysis suggest that the combination of green and gray strategies may still offer further potential for robustness.

Consequence management of chemical intrusion in water distribution networks under inexact scenarios

The US Environmental Protection Agency (EPA)'s Response Protocol Toolbox provides a list of recommendations on actions that may be taken to minimize the potential threats to public health following a contamination threat. This protocol comprises three steps: (1) detection of contaminant presence, (2) source identification and (3) consequence management. This paper intends to explore consequence management under source uncertainty, applying Minimize Maximum Regret (MMR) and Minimize Total Regret (MTR) approaches. An ant colony optimization algorithm is coupled with the EPANET network solver for structuring the MMR and MTR models to present a robust method for consequence management by selecting the best combination of hydrants and valves for isolation and contamination flushing out of the system. The proposed models are applied to network number 3 of EPANET to present its effectiveness and capabilities in developing effective consequence management strategies.

Performance analysis of opportunistic spectrum access based on partially observable Markov decision process

To maximize the throughout of AD hoc system with opportunistic spectrum access (OSA) among multiple secondary users is analyzed. Two distributed learning and allocation schemes are studied by using greedy and random approach, which can maximize the cognitive system throughput or equivalently minimize the total regret in distributed learning and allocation. The simulation results show that the performance of OSA in AD hoc system are affected by package arrival rate and the number of second users. When the second users are less than six, or the package arrival rate is less than 0.06, the throughput of greedy approach is higher than that of random approach, and greedy approach is recommended to be adopted in the design of OSA in AD hoc system.

Distributed Algorithms for Learning and Cognitive Medium Access with Logarithmic Regret

The problem of distributed learning and channel access is considered in a cognitive network with multiple secondary users. The availability statistics of the channels are initially unknown to the secondary users and are estimated using sensing decisions. There is no explicit information exchange or prior agreement among the secondary users and sensing and access decisions are undertaken by them in a completely distributed manner. We propose policies for distributed learning and access which achieve order-optimal cognitive system throughput (number of successful secondary transmissions) under self play, i.e., when implemented at all the secondary users. Equivalently, our policies minimize the sum regret in distributed learning and access, which is the loss in secondary throughput due to learning and distributed access. For the scenario when the number of secondary users is known to the policy, we prove that the total regret is logarithmic in the number of transmission slots. This policy achieves order-optimal regret based on a logarithmic lower bound for regret under any uniformly-good learning and access policy. We then consider the case when the number of secondary users is fixed but unknown, and is estimated at each user through feedback. We propose a policy whose sum regret grows only slightly faster than logarithmic in the number of transmission slots.

Near-optimal Regret Bounds for Reinforcement Learning

For undiscounted reinforcement learning in Markov decision processes (MDPs) we consider the total regret of a learning algorithm with respect to an optimal policy. In order to describe the transiti...

A min-max regret approach to unbalanced bidding in construction

Unbalanced bidding is a method to benefit from uneven markup distribution among different items of project. As bidding process is applied for a project which is going to be done in future, most of parameters cannot be estimated accurately. Therefore, there are some uncertainties in bidding phase. Uncertainties in quantities of works play an important role for unit price determination in an unbalanced bidding model. Therefore, in this paper these uncertainties are considered, applying Minimize Maximum Regret (MMR) and Minimize Total Regret (MTR) approach in a discrete area where each scenario represents a set of possible quantities of works. Allocating interval scenario case for quantities of works seems to be more appropriate than discrete one. Thus, the unbalanced bidding model with interval quantities of works using MMR is proposed. To define manageable number of scenarios resulting from possible combinations of different unit prices in formulating the Min Max Regret (MMR) model as interval scenario case, a relaxation procedure is employed. In this approach, instead of considering all possible objective functions, a model which is called “Candidate Maximum Regret (CMR)” is applied to determine worst case scenarios. Models are applied to a hypothetical case example and the results are compared.

Play‐the‐winner rule and adaptive designs of clinical trials

In another paper, we have argued that the traditional randomized design of clinical trials is ethically infeasible in desperate medical situations and adaptive designs are morally required. We have also argued that in such situations, the appropriate designs must satisfy what we call the “Principle of interchangeability.” In this statistics paper, we show that the statistical model of bandit processes satisfies this principle of interchangeability. Moreover, we demonstrate that when such a model is used as an adaptive design, the total regret of successes lost is smaller when compared with simple randomization. We illustrate the results by the simple deterministic play‐the‐winner design.

The compromise criterion in MCDM: interpretation and sensitivity to the p parameter

In this paper we discuss the application of the compromise criterion to multicriteria decisionmaking and present an application taken from the field of architecture. The Lp (Minkowski) metric is used to rank a given set of alternatives according to their generalized distance from the ideal solution. We also examine the interpretation of the parameter p and the sensitivity of the results to its value. When p = 1, the compromise solution minimizes the total regret; when p is very large, the alternative selected minimizes very low performance as regards all the objectives. However, intermediate values of p which lack a clear linguistic interpretation imply a second level of compromise between the two extreme strategies. An empirical problem is used to illustrate the model. It concerns the selection of lightweight construction systems for the enclosure of outdoor swimming pools. Fifteen construction systems were evaluated and ranked on the basis of twenty-six different criteria. The sensitivity analysis exemp...