Linear Reward Research Articles

Admission Control in Multi-server Systems Under Binary Reward Structure

We study a multi-server queueing system where a customer is satisfied (and generates a unit revenue) if their queueing time is at most a given constant. If the queueing time of the admitted customer exceeds this constant, the customer gets served, but is unsatisfied and generates no revenue. Such queueing systems arise in the context of modeling service systems where excessive delays are of concern. A key challenge is how to design an admission control policy to maximize the number of satisfied customers per unit time in the long run, assuming that we can observe the number of customers in the system at any time. We call this the binary reward structure system and show that a threshold-type admission policy is optimal. The optimal threshold policy has to be computed numerically. Hence we propose a square-root admission policy to approximate the optimal admission control policy, and compare the performance of these two policies. We derive an analytical upper bound on the performance of optimal admission control policy by deriving an optimal admission policy assuming we have full information over the queueing time of the admitted customers. This is equivalent to a queueing system where customers abandon the queue (i.e., leave without service) if their queueing time exceeds the given constant. We demonstrate that the optimal policy that includes customer abandonment, or alternatively, the optimal policy under full information, the optimal threshold policy, and the square-root admission policy, all exhibit identical performance in the asymptotic regions of the parameter space. Our numerical results indicate that the worst optimality gap of the square-root admission policy is within 3.9% of the optimal revenue, and implementing the square-root admission policy in the observable queueing system leads to a revenue loss that is at most 5.6% of the maximum possible revenue rate in the full information system. We also compare the binary reward structure with the more common linear reward structure where the system incurs holding cost per unit queueing time per customer. In addition, we also show that the analysis based on queueing time is applicable to the system time as well.

A Multiplier Bootstrap Approach to Designing Robust Algorithms for Contextual Bandits.

Upper confidence bound (UCB)-based contextual bandit algorithms require one to know the tail property of the reward distribution. Unfortunately, such tail property is usually unknown or difficult to specify in real-world applications. Using a tail property heavier than the ground truth leads to a slow learning speed of the contextual bandit algorithm, while using a lighter one may cause the algorithm to diverge. To address this fundamental problem, we develop an estimator (evaluated from historical rewards) for the contextual bandit UCB based on the multiplier bootstrap technique. Our proposed estimator mitigates the problem of specifying a heavier tail property by adaptively converging to the ground truth contextual bandit UCB (i.e., eliminating the impact of the specified heavier tail property) with theoretical guarantees on the convergence. The design and convergence analysis of the proposed estimator is technically nontrivial. The proposed estimator is generic and it can be applied to improve a variety of UCB-based contextual bandit algorithms. To demonstrate the versatility of the proposed estimator, we apply it to improve the linear reward contextual bandit UCB (LinUCB) algorithm resulting in our bootstrapping LinUCB (BootLinUCB) algorithm. We prove that the BootLinUCB has a sublinear regret. We conduct extensive experiments on both synthetic dataset and real-world dataset from Yahoo! to validate the benefits of our proposed estimator in reducing regret and the superior performance of BootLinUCB over the latest baseline.

A learning-based approach for video streaming over fluctuating networks with limited playback buffers

In recent times, the surge in applications like social networking and online learning platforms has led to a substantial rise in on-demand video streaming, thus ensuring a seamless user experience holds paramount importance, particularly given the dynamic nature of network conditions. Moreover, numerous service providers are embracing smaller buffer sizes, aiming to reduce bandwidth inefficiencies due to the possibility of users ending video sessions prematurely. However, this transition presents a significant challenge for conventional adaptive bitrate (ABR) algorithms, as they grapple with the task of harmonizing low stalling time, high playback bitrate, and the constraint of a minimized buffer size. In this study, we introduce a novel ABR approach, L2-ABR, which leverages self-play reinforcement learning to address these complexities. Unlike conventional reward-engineering learning-based ABR strategies that update gradients to maximize linear reward functions, L2-ABR treats video streaming as a fundamental objective and trains neural networks (NNs) with explicit requirements tailored to video streaming with small playback buffers. Through an extensive series of trace-driven experiments, we showcase that L2-ABR outperforms existing methods, effectively striking a balance between buffer management and network scenarios without inducing excessive buffer under-runs or overflows. Compared to existing ABR schemes, our method reduces undesirable buffer events, including rebuffering events and buffer full events, while improving the average QoE by up to 71.88% and 75.25% over the HSDPA and FCC datasets, respectively.

Distributed Task Management in Fog Computing: A Socially Concave Bandit Game

Fog computing leverages the task offloading capabilities at the network’s edge to improve efficiency and enable swift responses to application demands. However, the design of task allocation strategies in a fog computing network is still challenging because of the heterogeneity of fog nodes and uncertainties in system dynamics. We formulate the distributed task allocation problem as a social-concave game with bandit feedback and show that the game has a unique Nash equilibrium, which is implementable using no-regret learning strategies (regret with sublinear growth). We then develop two no-regret online decision-making strategies. One strategy, namely bandit gradient ascent with momentum, is an online convex optimization algorithm with bandit feedback. The other strategy, Lipschitz bandit with initialization, is an EXP3 multi-armed bandit algorithm. We establish regret bounds for both strategies and analyze their convergence characteristics. Moreover, we compare the proposed strategies with an allocation strategy named learning with linear rewards. Theoretical-and numerical analysis shows the superior performance of the proposed strategies for efficient task allocation compared to the state-of-the-art methods.

Evaluating zero-determinant strategies’ effects on cooperation and conflict resolution in repeated games

This paper discusses the importance of using rationality and freedom to serve oneself and cooperate with others. It highlights the possibility of the prisoner’s dilemma as a model for collaboration in a collaborative scenario. It answers the question of how zero-determinant strategies can be used to impose a one-sided linear reward relationship between a player’s payoff and his opponent’s payoff. Researchers focus on manipulative zero-determinant strategies and their impact on evolutionary gaming outcomes in large populations. In our work, we elicit the necessary and sufficient conditions for an actionable reward relationship in multi-player social dilemmas with a limited number of expected rounds. The study explores the concept of rewarded relationships and their role in achieving desired results. We provide insights into strategies players can use to maximize their gains in these dilemmas. In general, the study emphasizes the importance of using game theory to address complex problems in various fields.

Performance analysis of reinforcement learning algorithms on intelligent closed-loop control on fluid flow and convective heat transfer

This paper investigates the performance of several most popular deep reinforcement learning (DRL) algorithms applied to fluid flow and convective heat transfer systems, providing credible guidance and evaluation on their characteristics and performance. The studied algorithms are selected by considering the popularity, category, and advancement for guaranteeing the significance of the current study. The effectiveness and feasibility of all DRL algorithms are first demonstrated by studying a two-dimensional multi-heat-source cooling problem. Compared with the best manually optimized control, all DRL algorithms can find better control strategies that realize a further temperature reduction of 3–7 K. For problems with complex control objectives and environments, PPO (proximal policy optimization) shows an outstanding performance that accurately and dynamically constrains the oscillation of the solid temperature within 0.5 K around the target value, which is far beyond the capability of the manually optimized control. With the presented performance and the supplemented generalization test, the characteristic and specialty of the DRL algorithms are analyzed. The value-based methods have better training efficiency on simple cooling tasks with linear reward, while the policy-based methods show remarkable convergence on demanding tasks with nonlinear reward. Among the algorithms studied, the single-step PPO and prioritized experience replay deep Q-networks should be highlighted: the former has the advantage of considering multiple control targets and the latter obtains the best result in all generalization testing tasks. In addition, randomly resetting the environment is confirmed to be indispensable for the trained agent executing long-term control, which is strongly recommended to be included in follow-up studies.

A Fair Incentive Scheme for Community Health Workers

Community health workers (CHWs) play a crucial role in the last mile delivery of essential health services to underserved populations in low-income countries. Many nongovernmental organizations (NGOs) provide training and support to enable CHWs to deliver health services to their communities, with no charge to the recipients of the services. This includes monetary compensation for the work that CHWs perform, which is broken down into a series of well defined tasks. In this work, we partner with a NGO D-Tree International to design a fair monetary compensation scheme for tasks performed by CHWs in the semi-autonomous region of Zanzibar in Tanzania, Africa. In consultation with stakeholders, we interpret fairness as the equal opportunity to earn, which means that each CHW has the opportunity to earn roughly the same total payment over a given T month period, if the CHW reacts to the incentive scheme almost rationally. We model this problem as a reward design problem for a Markov Decision Process (MDP) formulation for the CHWs’ earning. There is a need for the mechanism to be simple so that it is understood by the CHWs, thus, we explore linear and piecewise linear rewards in the CHWs’ measured units of work. We solve this design problem via a novel policy-reward gradient result. Our experiments using two real world parameters from the ground provide evidence of reasonable incentive output by our scheme.

Learning Noise-Induced Reward Functions for Surpassing Demonstrations in Imitation Learning

Imitation learning (IL) has recently shown impressive performance in training a reinforcement learning agent with human demonstrations, eliminating the difficulty of designing elaborate reward functions in complex environments. However, most IL methods work under the assumption of the optimality of the demonstrations and thus cannot learn policies to surpass the demonstrators. Some methods have been investigated to obtain better-than-demonstration (BD) performance with inner human feedback or preference labels. In this paper, we propose a method to learn rewards from suboptimal demonstrations via a weighted preference learning technique (LERP). Specifically, we first formulate the suboptimality of demonstrations as the inaccurate estimation of rewards. The inaccuracy is modeled with a reward noise random variable following the Gumbel distribution. Moreover, we derive an upper bound of the expected return with different noise coefficients and propose a theorem to surpass the demonstrations. Unlike existing literature, our analysis does not depend on the linear reward constraint. Consequently, we develop a BD model with a weighted preference learning technique. Experimental results on continuous control and high-dimensional discrete control tasks show the superiority of our LERP method over other state-of-the-art BD methods.

Provably Efficient Reinforcement Learning with Linear Function Approximation

Modern reinforcement learning (RL) is commonly applied to practical problems with an enormous number of states, where function approximation must be deployed to approximate either the value function or the policy. The introduction of function approximation raises a fundamental set of challenges involving computational and statistical efficiency, especially given the need to manage the exploration/exploitation trade-off. As a result, a core RL question remains open: how can we design provably efficient RL algorithms that incorporate function approximation? This question persists even in a basic setting with linear dynamics and linear rewards, for which only linear function approximation is needed. This paper presents the first provable RL algorithm with both polynomial run time and polynomial sample complexity in this linear setting, without requiring a “simulator” or additional assumptions. Concretely, we prove that an optimistic modification of least-squares value iteration—a classical algorithm frequently studied in the linear setting—achieves [Formula: see text] regret, where d is the ambient dimension of feature space, H is the length of each episode, and T is the total number of steps. Importantly, such regret is independent of the number of states and actions. Funding: This work was supported by the Defense Advanced Research Projects Agency program on Lifelong Learning Machines.

ANFIS and metaheuristic optimization for strategic joining policy with re-attempt and vacation

This research work deals with the queueing analysis and strategic behavior of primitive users (PUs) and ordinary users (OUs) in cognitive radio system (CRS). Whenever fusion center (FC) is occupied, the ordinary users must decide whether to join a retrial orbit and make re-attempts or depart from the system. The OUs residing in the retrial orbit try to access the spectrum through FC. The FC becomes inactive and considered on vacation in case if there are no users in the CRS. Returning back from the vacation, the FC renders service to the primitive users. The service of ordinary users (OUs) may be interrupted due to breakdown of FC. The probability generating function approach is used to derive the performance indices related to the queueing characterization. The joining strategic behavior of the primitive as well as ordinary users has been investigated by considering a linear reward cost function. The social welfare by formulating the cooperative strategy is studied. The hybrid soft computing technique viz. adaptive network fuzzy inference system (ANFIS) is employed for the numerical simulation and sensitivity analysis of various system metrics. The optimal service rates are determined by minimizing the total cost involved in various activities using the quasi-Newton method (QNM) and metaheuristic algorithms.

Solving Two-Person Zero-Sum Stochastic Games With Incomplete Information Using Learning Automata With Artificial Barriers.

Learning automata (LA) with artificially absorbing barriers was a completely new horizon of research in the 1980s (Oommen, 1986). These new machines yielded properties that were previously unknown. More recently, absorbing barriers have been introduced in continuous estimator algorithms so that the proofs could follow a martingale property, as opposed to monotonicity (Zhang et al., 2014), (Zhang et al., 2015). However, the applications of LA with artificial barriers are almost nonexistent. In that regard, this article is pioneering in that it provides effective and accurate solutions to an extremely complex application domain, namely that of solving two-person zero-sum stochastic games that are provided with incomplete information. LA have been previously used (Sastry et al., 1994) to design algorithms capable of converging to the game's Nash equilibrium under limited information. Those algorithms have focused on the case where the saddle point of the game exists in a pure strategy. However, the majority of the LA algorithms used for games are absorbing in the probability simplex space, and thus, they converge to an exclusive choice of a single action. These LA are thus unable to converge to other mixed Nash equilibria when the game possesses no saddle point for a pure strategy. The pioneering contribution of this article is that we propose an LA solution that is able to converge to an optimal mixed Nash equilibrium even though there may be no saddle point when a pure strategy is invoked. The scheme, being of the linear reward-inaction ( LR-I ) paradigm, is in and of itself, absorbing. However, by incorporating artificial barriers, we prevent it from being "stuck" or getting absorbed in pure strategies. Unlike the linear reward- ϵ penalty ( LR-ϵP ) scheme proposed by Lakshmivarahan and Narendra almost four decades ago, our new scheme achieves the same goal with much less parameter tuning and in a more elegant manner. This article includes the nontrial proofs of the theoretical results characterizing our scheme and also contains experimental verification that confirms our theoretical findings.

Asymmetric and adaptive reward coding via normalized reinforcement learning.

Learning is widely modeled in psychology, neuroscience, and computer science by prediction error-guided reinforcement learning (RL) algorithms. While standard RL assumes linear reward functions, reward-related neural activity is a saturating, nonlinear function of reward; however, the computational and behavioral implications of nonlinear RL are unknown. Here, we show that nonlinear RL incorporating the canonical divisive normalization computation introduces an intrinsic and tunable asymmetry in prediction error coding. At the behavioral level, this asymmetry explains empirical variability in risk preferences typically attributed to asymmetric learning rates. At the neural level, diversity in asymmetries provides a computational mechanism for recently proposed theories of distributional RL, allowing the brain to learn the full probability distribution of future rewards. This behavioral and computational flexibility argues for an incorporation of biologically valid value functions in computational models of learning and decision-making.

Joint Optimization of Edge Computing Resource Pricing and Wireless Caching for Blockchain-Driven Networks

In wireless communication networks, traffics of base stations (BS) can be effectively offloaded by sharing the cached content between mobile devices through wireless device-to-device (D2D) communication. To encourage the content caching of D2D users, a blockchain incentive scheme is proposed; the game theory is introduced to optimize the benefits for both the edge computing server (ECS) and D2D users in the blockchain network. In Stackelberg game, D2D users act as followers. After the successful implementation of the cache strategy, D2D users are rewarded by the system, which is a new block mining process. In this process, the required computing power is provided by the ECS. Therefore, D2D cache users need to pay the cost to purchase computing power from the ECS. As the game leader, the ECS prices the computing resources, and D2D users adjust their caching strategies based on the price. In this paper, we propose four algorithms based on different pricing methods and different cache reward methods. Simulations are performed to compare the cache quality and cache content dispersion of the four algorithms. The results show that the uniform pricing scheme with a linear reward relationship is more suitable for the scenarios that require higher cache quality, and the discriminatory pricing scheme with a nonlinear reward relationship is more suitable for the scenarios that require more evenly cache content distribution.

A Reinforcement Learning-Based Dynamic Clustering Algorithm for Compressive Data Gathering in Wireless Sensor Networks

Compressive data gathering (CDG) is an effective technique to handle large amounts of data transmissions in resource-constrained wireless sensor networks (WSNs). However, CDG with static clustering cannot adapt to time-varying environments in WSNs. In this paper, a reinforcement learning-based dynamic clustering algorithm (RLDCA) for CDG in WSNs is proposed. It is a dynamic and adaptive clustering method aiming to further reduce data transmissions and energy consumption in WSNs. Sensor nodes act as reinforcement learning (RL) agents which can observe the environment and dynamically select a cluster to join in. These RL agents are instructed by a well-designed reward scheme to join a cluster with strong data correlation and proper distance. It is also a distributed and lightweight learning method. All agents are independent and operate in parallel. Additional overheads introduced by RL are lightweight. Computations of a linear reward function and a few comparison operations are needed. It is implementable in WSNs. Simulations performed in MATLAB validate the effectiveness of the proposed method and simulation results show that the proposed algorithm achieves the desired effect as well as fine convergence. It decreases data transmissions by 16.6% and 54.4% and energy consumption by 6% and 29%, respectively, compared to the two contrastive schemes.

Distributed Reinforcement Learning for the Management of a Smart Grid Interconnecting Independent Prosumers

In the context of an eco-responsible production and distribution of electrical energy at the local scale of an urban territory, we consider a smart grid as a system interconnecting different prosumers, which all retain their decision-making autonomy and defend their own interests in a comprehensive system where the rules, accepted by all, encourage virtuous behavior. In this paper, we present and analyze a model and a management method for smart grids that is shared between different kinds of independent actors, who respect their own interests, and that encourages each actor to behavior that allows, as much as possible, an energy independence of the smart grid from external energy suppliers. We consider here a game theory model, in which each actor of the smart grid is a player, and we investigate distributed machine-learning algorithms to allow decision-making, thus, leading the game to converge to stable situations, in particular to a Nash equilibrium. We propose a Linear Reward Inaction algorithm that achieves Nash equilibria most of the time, both for a single time slot and across time, allowing the smart grid to maximize its energy independence from external energy suppliers.

Twin Delayed Deep Deterministic Policy Gradient-Based Target Tracking for Unmanned Aerial Vehicle With Achievement Rewarding and Multistage Training

Target tracking using an unmanned aerial vehicle (UAV) is a challenging robotic problem. It requires handling a high level of nonlinearity and dynamics. Model-free control effectively handles the uncertain nature of the problem, and reinforcement learning (RL)-based approaches are a good candidate for solving this problem. In this article, the Twin Delayed Deep Deterministic Policy Gradient Algorithm (TD3), as recent and composite architecture of RL, was explored as a tracking agent for the UAV-based target tracking problem. Several improvements on the original TD3 were also performed. First, the proportional-differential controller was used to boost the exploration of the TD3 in training. Second, a novel reward formulation for the UAV-based target tracking enabled a careful combination of the various dynamic variables in the reward functions. This was accomplished by incorporating two exponential functions to limit the effect of velocity and acceleration to prevent the deformation in the policy function approximation. In addition, the concept of multistage training based on the dynamic variables was proposed as an opposing concept to one-stage combinatory training. Third, an enhancement of the rewarding function by including piecewise decomposition was used to enable more stable learning behaviour of the policy and move out from the linear reward to the achievement formula. The training was conducted based on fixed target tracking followed by moving target tracking. The flight testing was conducted based on three types of target trajectories: fixed, square, and blinking. The multistage training achieved the best performance with both exponential and achievement rewarding for the fixed trained agent with the fixed and square moving target and for the combined agent with both exponential and achievement rewarding for a fixed trained agent in the case of a blinking target. With respect to the traditional proportional differential controller, the maximum error reduction rate is 86%. The developed achievement rewarding and the multistage training opens the door to various applications of RL in target tracking.

Symbolic Dynamic Programming for Continuous State and Action MDPs

Many real-world decision-theoretic planning problemsare naturally modeled using both continuous state andaction (CSA) spaces, yet little work has provided ex-act solutions for the case of continuous actions. Inthis work, we propose a symbolic dynamic program-ming (SDP) solution to obtain the optimal closed-formvalue function and policy for CSA-MDPs with mul-tivariate continuous state and actions, discrete noise,piecewise linear dynamics, and piecewise linear (or re-stricted piecewise quadratic) reward. Our key contribu-tion over previous SDP work is to show how the contin-uous action maximization step in the dynamic program-ming backup can be evaluated optimally and symboli-cally — a task which amounts to symbolic constrainedoptimization subject to unknown state parameters; wefurther integrate this technique to work with an efficientand compact data structure for SDP — the extendedalgebraic decision diagram (XADD). We demonstrateempirical results on a didactic nonlinear planning exam-ple and two domains from operations research to showthe first automated exact solution to these problems.

Reinforcement Learning and Particle Swarm Optimization Supporting Real-Time Rescue Assignments for Multiple Autonomous Underwater Vehicles

Rescue assignments strategy are crucial for multiple Autonomous Underwater Vehicle (multi-AUV) systems in three dimensional (3-D) complex underwater environments. Considering the requirements of rescue missions, multi-AUV systems need to be cost-effective, fast-rescuing, and less concerned about the relationship between rescue missions. The real-time rescue plays a vital role in the multi-AUV system with the characteristics mentioned above. In this paper, we propose an efficient Reward acting on Reinforcement Learning and Particle Swarm Optimization (R-RLPSO), to provide a strategy of real-time rescue assignment for the multi-AUV system in the 3-D underwater environment. This strategy consists of the following three parts. Firstly, we present a reward-based real-time rescue assignment algorithm. Secondly, we propose an Attraction Rescue Area containing a Rescue Area. For the waypoints in each Attraction Rescue Area, the reward is calculated by a linear reward function. Thirdly, to speed up the convergence of the R-RLPSO and mark the rescue states of Attraction Rescue Area and rescue area, we develop a Reward Coefficient based on the reward of all Attraction Rescue Areas and Rescue Areas. Finally, simulation results show that the system based on R-RLPSO is more cost-effective and time-saving than that of based on comparison algorithms ISOM and IACO.

The Restless Hidden Markov Bandit With Linear Rewards and Side Information

In this paper we present a model for the hidden Markovian bandit problem with linear rewards. As opposed to current work on Markovian bandits, we do not assume that the state is known to the decision maker before making the decision. Furthermore, we assume structural side information where the decision maker knows in advance that there are two types of hidden states; one is common to all arms and evolves according to a Markovian distribution, and the other is unique to each arm and is distributed according to an i.i.d. process that is unique to each arm. We present an algorithm and regret analysis to this problem. Surprisingly, we can recover the hidden states and maintain logarithmic regret in the case of a convex polytope action set. Furthermore, we show that the structural side information leads to expected regret that does not depend on the number of extreme points in the action space. Therefore, we obtain practical solutions even in high dimensional problems.

Error Noted in “Robust Contract Designs: Linear Contracts and Moral Hazard” by Yu and Kong (2020)

One of the main results of “Robust Contract Designs: Linear Contracts and Moral Hazard” by Yu and Kong (2020) is Proposition 4, which states that the optimal robust contract with a piecewise linear concave agent utility only consists of progressive fixed payments and linear rewards with progressive commission rates. In this note, we construct a counterexample showing that this result is wrong.