Average Reward Reinforcement Learning Research Articles

Multi-provider NFV network service delegation via average reward reinforcement learning

In multi-provider 5G/6G networks, service delegation enables administrative domains to federate in provisioning NFV network services. Admission control in selecting the appropriate domain for service deployment, without prior knowledge of service requests’ statistical distributions, is fundamental to maximize average profit. This paper analyzes a general federation contract model for service delegation in various ways. First, under the assumption of known system dynamics, we obtain the theoretically optimal performance bound by formulating the admission control problem as an infinite-horizon Markov decision process (MDP) and solving it through dynamic programming, which is used as a benchmark to evaluate practical solutions. Second, we apply Reinforcement Learning (RL) to practically tackle the problem when the arrival and departure rates are not known. For the first time in this context, we analyze the performance of the widely used Q-Learning algorithm, and prove as it maximizes the discounted rewards, it is not an efficient solution due to its sensitivity to the discount factor. Then, we propose the average reward reinforcement learning approach (named “R-Learning”) to find the policy that directly maximizes the average profit. Finally, we evaluate different solutions through extensive simulations and experimentally using the 5Growth management and orchestration platform. Results confirm that the proposed R-Learning solution always outperforms Q-Learning and the greedy policies. Furthermore, while there is at most a 9% optimality gap in the ideal simulation environment, it competes with the MDP solution in the experimental assessment.

Performance-gated deliberation: A context-adapted strategy in which urgency is opportunity cost.

Finding the right amount of deliberation, between insufficient and excessive, is a hard decision making problem that depends on the value we place on our time. Average-reward, putatively encoded by tonic dopamine, serves in existing reinforcement learning theory as the opportunity cost of time, including deliberation time. Importantly, this cost can itself vary with the environmental context and is not trivial to estimate. Here, we propose how the opportunity cost of deliberation can be estimated adaptively on multiple timescales to account for non-stationary contextual factors. We use it in a simple decision-making heuristic based on average-reward reinforcement learning (AR-RL) that we call Performance-Gated Deliberation (PGD). We propose PGD as a strategy used by animals wherein deliberation cost is implemented directly as urgency, a previously characterized neural signal effectively controlling the speed of the decision-making process. We show PGD outperforms AR-RL solutions in explaining behaviour and urgency of non-human primates in a context-varying random walk prediction task and is consistent with relative performance and urgency in a context-varying random dot motion task. We make readily testable predictions for both neural activity and behaviour.

Temporal-Logic-Based Reward Shaping for Continuing Reinforcement Learning Tasks

In continuing tasks, average-reward reinforcement learning may be a more appropriate problem formulation than the more common discounted reward formulation. As usual, learning an optimal policy in this setting typically requires a large amount of training experiences. Reward shaping is a common approach for incorporating domain knowledge into reinforcement learning in order to speed up convergence to an optimal policy. However, to the best of our knowledge, the theoretical properties of reward shaping have thus far only been established in the discounted setting. This paper presents the first reward shaping framework for average-reward learning and proves that, under standard assumptions, the optimal policy under the original reward function can be recovered. In order to avoid the need for manual construction of the shaping function, we introduce a method for utilizing domain knowledge expressed as a temporal logic formula. The formula is automatically translated to a shaping function that provides additional reward throughout the learning process. We evaluate the proposed method on three continuing tasks. In all cases, shaping speeds up the average-reward learning rate without any reduction in the performance of the learned policy compared to relevant baselines.

Improving selection strategies in zeroth-level classifier systems based on average reward reinforcement learning

As a genetics-based machine learning technique, zeroth-level classifier system (ZCS) based on average reward reinforcement learning (ZCSAR) evolves solutions to optimize average reward per time step. However, initial experimental results have shown that, in some cases, the performance of ZCSAR oscillates heavily during the learning period, or cannot reach the optimum during the testing period. In this paper, we modify the selection strategies in ZCSAR to improve its performance, under conditions of minimal changes of ZCSAR. The proposed selection strategies take tournament selection method to choose parents in Genetic Algorithm (GA), and take roulette wheel selection method to choose actions in match set and to choose classifiers for deletion in both GA and covering. Experimental results show that ZCSAR with the new selection strategies can evolve more promising solutions with enough parameter independence, and also with slighter oscillation during the learning period.

Tournament Selection in Zeroth-Level Classifier Systems Based On Average Reward Reinforcement Learning

As a genetics-based machine learning technique, zeroth-level classifier system (ZCS) is based on a discounted reward reinforcement learning algorithm, bucket-brigade algorithm, which optimizes the discounted total reward received by an agent but is not suitable for all multi-step problems, especially large-size ones. There are some undiscounted reinforcement learning methods available, such as R-learning, which optimize the average reward per time step. In this paper, R-learning is used as the reinforcement learning employed by ZCS, to replace its discounted reward reinforcement learning approach, and tournament selection is used to replace roulette wheel selection in ZCS. The modification results in classifier systems that can support long action chains, and thus is able to solve large multi-step problems.

Efficient Average Reward Reinforcement Learning Using Constant Shifting Values

There are two classes of average reward reinforcement learning (RL) algorithms: model-based ones that explicitly maintain MDP models and model-free ones that do not learn such models. Though model-free algorithms are known to be more efficient, they often cannot converge to optimal policies due to the perturbation of parameters. In this paper, a novel model-free algorithm is proposed, which makes use of constant shifting values (CSVs) estimated from prior knowledge. To encourage exploration during the learning process, the algorithm constantly subtracts the CSV from the rewards. A terminating condition is proposed to handle the unboundedness of Q-values caused by such substraction. The convergence of the proposed algorithm is proved under very mild assumptions. Furthermore, linear function approximation is investigated to generalize our method to handle large-scale tasks. Extensive experiments on representative MDPs and the popular game Tetris show that the proposed algorithms significantly outperform the state-of-the-art ones.

Order acceptance policy in Make-to-Order manufacturing based on average-reward reinforcement learning

From the perspective of revenue management,a new approach for order acceptance under uncertainty in Make-to-Oder(MTO) manufacturing using average-reward reinforcement learning was proposed.In order to maximize the average expected revenue,the proposed approach took order types and different combinations of price and leadtime as criteria for the classification of the system states based on multi-level pricing mechanism.The simulation results show that the proposed algorithm has learning and selective ability to accept the order.Comparisons made with other order acceptance policies show the effectiveness of the proposed algorithm in average revenue,accepted order types,and adaptability.

Learning classifier system with average reward reinforcement learning

In the family of Learning Classifier Systems, the classifier system XCS is most widely used and investigated. However, the standard XCS has difficulties solving large multi-step problems, where long action chains are needed to get delayed rewards. Up to the present, the reinforcement learning technique in XCS has been based on Q-learning, which optimizes the discounted total reward received by an agent but tends to limit the length of action chains. However, there are some undiscounted reinforcement learning methods available, such as R-learning and average reward reinforcement learning in general, which optimize the average reward per time step. In this paper, R-learning is used as the reinforcement learning employed by XCS, to replace Q-learning. The modification results in a classifier system that is rapid and able to solve large maze problems. In addition, it produces uniformly spaced payoff levels, which can support long action chains and thus effectively prevent the occurrence of overgeneralization.

R([formula omitted]) imitation learning for automatic generation control of interconnected power grids

The goal of average reward reinforcement learning is to maximize the long-term average rewards of a generic system. This coincides with the design objective of the control performance standards (CPS) which were established to improve the long-term performance of an automatic generation controller (AGC) used for real-time control of interconnected power systems. In this paper, a novel R(λ) imitation learning (R(λ)IL) method based on the average reward optimality criterion is presented to develop an optimal AGC under the CPS. This R(λ)IL-based AGC can operate online in real-time with high CPS compliances and fast convergence rate in the imitation pre-learning process. Its capability to learn the control behaviors of the existing AGC by observing system variations enable it to overcome the serious defect in the applicability of conventional RL controllers, in which an accurate power system model is required for the offline pre-learning process, and significantly enhance the learning efficiency and control performance for power generation control in various power system operation scenarios.

Robustness of optimal channel reservation using handover prediction in multiservice wireless networks

The aim of our study is to obtain theoretical limits for the gain that can be expected when using handover prediction and to determine the sensitivity of the system performance against different parameters. We apply an average-reward reinforcement learning approach based on afterstates to the design of optimal admission control policies in mobile multimedia cellular networks where predictive information related to the occurrence of future handovers is available. We consider a type of predictor that labels active mobile terminals in the cell neighborhood a fixed amount of time before handovers are predicted to occur, which we call the anticipation time. The admission controller exploits this information to reserve resources efficiently. We show that there exists an optimum value for the anticipation time at which the highest performance gain is obtained. Although the optimum anticipation time depends on system parameters, we find that its value changes very little when the system parameters vary within a reasonable range. We also find that, in terms of system performance, deploying prediction is always advantageous when compared to a system without prediction, even when the system parameters are estimated with poor precision.

Minimizing mean weighted tardiness in unrelated parallel machine scheduling with reinforcement learning

We address an unrelated parallel machine scheduling problem with R-learning, an average-reward reinforcement learning (RL) method. Different types of jobs dynamically arrive in independent Poisson processes. Thus the arrival time and the due date of each job are stochastic. We convert the scheduling problems into RL problems by constructing elaborate state features, actions, and the reward function. The state features and actions are defined fully utilizing prior domain knowledge. Minimizing the reward per decision time step is equivalent to minimizing the schedule objective, i.e. mean weighted tardiness. We apply an on-line R-learning algorithm with function approximation to solve the RL problems. Computational experiments demonstrate that R-learning learns an optimal or near-optimal policy in a dynamic environment from experience and outperforms four effective heuristic priority rules (i.e. WSPT, WMDD, ATC and WCOVERT) in all test problems.

Learning and adaptation of a policy for dynamic order acceptance in make-to-order manufacturing

Order acceptance under uncertainty is a critical decision-making problem at the interface between customer relationship management and production planning of order-driven manufacturing systems. In this work, a novel approach for simulation-based development and on-line adaptation of a policy for dynamic order acceptance under uncertainty in make-to-order manufacturing using average-reward reinforcement learning is proposed. Locally weighted regression is used to generalize the gain value of accepting or rejecting similar orders regarding attributes such as product mix, price, size and due date. The order acceptance policy is learned by classifying an arriving order as belonging either to the acceptance set or to the rejection set. For exploitation, only orders in the acceptance set must be chosen for shop-floor scheduling. For exploration some orders from the rejection set are also considered as candidates for acceptance. Comparisons made with different order acceptance heuristics highlight the effectiveness of the proposed ARLOA algorithm to maximize the average revenue obtained per unit cost of installed capacity whilst quickly responding to unknown variations in order arrival rates and attributes.

A New QoS Provisioning Method for Adaptive Multimedia in Wireless Networks

Future wireless networks are designed to support adaptive multimedia by controlling individual ongoing flows to increase or decrease their bandwidths in response to changes in traffic load. There is growing interest in quality-of-service (QoS) provisioning under this adaptive multimedia framework, in which a bandwidth adaptation algorithm needs to be used in conjunction with the call admission control algorithm. This paper presents a novel method for QoS provisioning via average reward reinforcement learning in conjunction with stochastic approximation, which can maximize the network revenue subject to several predetermined QoS constraints. Unlike other model-based algorithms (e.g., linear programming), our scheme does not require explicit state transition probabilities, and therefore, the assumptions behind the underlying system model are more realistic than those in previous schemes. In addition, when we consider the status of neighboring cells, the proposed scheme can dynamically adapt to changes in traffic condition. Moreover, the algorithm can control the bandwidth adaptation frequency effectively by accounting for the cost of bandwidth switching in the model. The effectiveness of the proposed approach is demonstrated using simulation results in adaptive multimedia wireless networks.

Tonic dopamine: opportunity costs and the control of response vigor

Dopamine neurotransmission has long been known to exert a powerful influence over the vigor, strength, or rate of responding. However, there exists no clear understanding of the computational foundation for this effect; predominant accounts of dopamine's computational function focus on a role for phasic dopamine in controlling the discrete selection between different actions and have nothing to say about response vigor or indeed the free-operant tasks in which it is typically measured. We seek to accommodate free-operant behavioral tasks within the realm of models of optimal control and thereby capture how dopaminergic and motivational manipulations affect response vigor. We construct an average reward reinforcement learning model in which subjects choose both which action to perform and also the latency with which to perform it. Optimal control balances the costs of acting quickly against the benefits of getting reward earlier and thereby chooses a best response latency. In this framework, the long-run average rate of reward plays a key role as an opportunity cost and mediates motivational influences on rates and vigor of responding. We review evidence suggesting that the average reward rate is reported by tonic levels of dopamine putatively in the nucleus accumbens. Our extension of reinforcement learning models to free-operant tasks unites psychologically and computationally inspired ideas about the role of tonic dopamine in striatum, explaining from a normative point of view why higher levels of dopamine might be associated with more vigorous responding.

Solving Semi-Markov Decision Problems Using Average Reward Reinforcement Learning

A large class of problems of sequential decision making under uncertainty, of which the underlying probability structure is a Markov process, can be modeled as stochastic dynamic programs (referred to, in general, as Markov decision problems or MDPs). However, the computational complexity of the classical MDP algorithms, such as value iteration and policy iteration, is prohibitive and can grow intractably with the size of the problem and its related data. Furthermore, these techniques require for each action the one step transition probability and reward matrices, and obtaining these is often unrealistic for large and complex systems. Recently, there has been much interest in a simulation-based stochastic approximation framework called reinforcement learning (RL), for computing near optimal policies for MDPs. RL has been successfully applied to very large problems, such as elevator scheduling, and dynamic channel allocation of cellular telephone systems. In this paper, we extend RL to a more general class of decision tasks that are referred to as semi-Markov decision problems (SMDPs). In particular, we focus on SMDPs under the average-reward criterion. We present a new model-free RL algorithm called SMART (Semi-Markov Average Reward Technique). We present a detailed study of this algorithm on a combinatorially large problem of determining the optimal preventive maintenance schedule of a production inventory system. Numerical results from both the theoretical model and the RL algorithm are presented and compared.

Model-based average reward reinforcement learning

Reinforcement Learning (RL) is the study of programs that improve their performance by receiving rewards and punishments from the environment. Most RL methods optimize the discounted total reward received by an agent, while, in many domains, the natural criterion is to optimize the average reward per time step. In this paper, we introduce a model-based Averagereward Reinforcement Learning method called H-learning and show that it converges more quickly and robustly than its discounted counterpart in the domain of scheduling a simulated Automatic Guided Vehicle (AGV). We also introduce a version of H-learning that automatically explores the unexplored parts of the state space, while always choosing greedy actions with respect to the current value function. We show that this “Auto-exploratory H-Learning” performs better than the previously studied exploration strategies. To scale H-learning to larger state spaces, we extend it to learn action models and reward functions in the form of dynamic Bayesian networks, and approximate its value function using local linear regression. We show that both of these extensions are effective in significantly reducing the space requirement of H-learning and making it converge faster in some AGV scheduling tasks.

Average reward reinforcement learning: Foundations, algorithms, and empirical results

This paper presents a detailed study of average reward reinforcement learning, an undiscounted optimality framework that is more appropriate for cyclical tasks than the much better studied discounted framework. A wide spectrum of average reward algorithms are described, ranging from synchronous dynamic programming methods to several (provably convergent) asynchronous algorithms from optimal control and learning automata. A general sensitive discount optimality metric calledn-discount-optimality is introduced, and used to compare the various algorithms. The overview identifies a key similarity across several asynchronous algorithms that is crucial to their convergence, namely independent estimation of the average reward and the relative values. The overview also uncovers a surprising limitation shared by the different algorithms while several algorithms can provably generategain-optimal policies that maximize average reward, none of them can reliably filter these to producebias-optimal (orT-optimal) policies that also maximize the finite reward to absorbing goal states. This paper also presents a detailed empirical study of R-learning, an average reward reinforcement learning method, using two empirical testbeds: a stochastic grid world domain and a simulated robot environment. A detailed sensitivity analysis of R-learning is carried out to test its dependence on learning rates and exploration levels. The results suggest that R-learning is quite sensitive to exploration strategies and can fall into sub-optimal limit cycles. The performance of R-learning is also compared with that of Q-learning, the best studied discounted RL method. Here, the results suggest that R-learning can be fine-tuned to give better performance than Q-learning in both domains.