Reward Sequence Research Articles

APPROXIMATION TO OPTIMAL STOPPING RULES FOR WEIBULL RANDOM VARIABLES WITH UNKNOWN SCALE PARAMETER

Let $X_1, X_2, \cdots, X_n, \cdots$ be independent, identically distributed Weibull random variables with an unknown scale parameter $\alpha$. If we define the reward sequence $Y_n = \max \{ X_1,X_2,\cdots,X_n \} - cn$ for $c \gt 0$, the optimal stopping rule for $Y_n$ depends on the unknown scale parameter $\alpha$. In this paper we propose an adaptive stopping rule that does not depend on the unknown scale parameter $\alpha$ and show that the difference between the optimal expected reward and the expected reward using the proposed adaptive stopping rule vanishes as $c$ goes to zero.

APPROXIMATION TO OPTIMAL STOPPING RULES FOR WEIBULL RANDOM VARIABLES WITH UNKNOWN SCALE PARAMETER

Let $X_1, X_2,\cdots, X_n,\cdots$ be independent, identically distributed Weibull random variables with an unknown scale parameter $\alpha$. If we define the reward sequence $Y_n=\max\{X_1,X_2,\cdots,X_n\}-cn$ for $c>0$, the optimal stopping rule for $Y_n$ depends on the unknown scale parameter $\alpha$. In this paper we propose an adaptive stopping rule that does not depend on the unknown scale parameter $\alpha$ and show that the difference between the optimal expected reward and the expected reward using the proposed adaptive stopping rule vanishes as $c$ goes to zero.

Cooperative Stopping Rules in Multivariate Problems

Consider a sequence of random vectors Y(1),…, Y(n) in ℛ d , with a known joint distribution, finite expectations, adapted to a filtration ℱ. For a given monotone function h : ℛ d → ℛ, a randomized stopping rule t ⋆ such that sup t h(E Y(t)) = h(E Y(t ⋆)) is desired. A full solution to this problem is given. In particular we show that there exists a vector of constants a = (a 1,…, a d ) such that the optimizing t ⋆ is optimal also for the one-dimensional optimal stopping problem, where the sequence of rewards is Z(1),…, Z(d) with Z(j) = . With symmetry assumptions the a vector will be (1,…, 1). The results are extended also to the infinite horizon case and are applied to obtain an explicit solution to the multivariate “house-selling problem.”

Performance measure sensitive congruences for Markovian process algebras

The modeling and analysis experience with process algebras has shown the necessity of extending them with priority, probabilistic internal/external choice, and time while preserving compositionality. The purpose of this paper is to make a further step by introducing a way to express performance measures, in order to allow the modeler to capture the QoS metrics of interest. We show that the standard technique of expressing stationary and transient performance measures as weighted sums of state probabilities and transition frequencies can be imported in the process algebra framework. Technically speaking, if we denote by n ∈ N the number of performance measures of interest, in this paper we define a family of extended Markovian process algebras with generative master–reactive slaves synchronization mechanism called EMPA gr n including probabilities, priorities, exponentially distributed durations, and sequences of rewards of length n. Then we show that the Markovian bisimulation equivalence ∼ MB n is a congruence for EMPA gr n which preserves the specified performance measures and we give a sound and complete axiomatization for finite EMPA gr n terms. Finally, we present a case study conducted with the software tool TwoTowers in which we contrast the average performance of a selection of distributed algorithms for mutual exclusion modeled with EMPA gr n .

Solving optimal stopping problems of linear diffusions by applying convolution approximations

We study how the convolution approximation of continuous mappings can be applied in solving optimal stopping problems of linear diffusions whenever the underlying payoff is not differentiable and the smooth fit principle does not necessarily apply. We construct a sequence of smooth reward functions converging uniformly on compacts to the original reward and, consequently, we derive a sequence of continuously differentiable (i.e. satisfying the smooth fit principle) value functions converging to the value of the original stopping problem.

APPROXIMATION TO OPTIMAL STOPPING RULES FOR GAMMA RANDOM VARIABLES WITH UNKNOWN LOCATION AND SCALE PARAMETERS

Let X 1, X 2, …, X n ,… be independent, identically distributed gamma random variables with unknown location and scale parameters. If we define the reward sequence Y n = max{X 1,X 2, …, X n } − cn, for c > 0, the optimal stopping rule for Y n depends on the unknown parameters. In this paper, we propose an adaptive stopping rule that does not depend on the parameters and show that the difference between the optimal expected reward and the expected reward using the proposed adaptive stopping rule vanishes as c goes to zero.

AVERAGE NUMBER OF EVENTS AND AVERAGE REWARD

In a renewal process, the interarrival times are independent and identically distributed, and in a renewal reward process, the pairs of reward and interarrival time are i.i.d. Many useful results hold for these processes. This paper relaxes the assumption of identical distributions while keeping the assumption of independence. This paper explores the properties of the mean average number of occurrence of events and the mean average reward on any finite time interval. The paper also discusses the limiting properties of these two quantities and extends many results from the renewal process and renewal reward process to the more general counting process and reward sequence.

Preference for sequences of rewards: further tests of a parallel discounting model

Brunner and Gibbon [Brunner, D., Gibbon, J., 1995. Anim. Behav. 50, 1627–1634] studied rats’ choices between sequences of equal numbers of rewards that differed in their temporal arrangement. Rats’ preferences were well accounted for by a parallel discounting model of aggregate food value in which the sequence value is the simple sum of the hyperbolically discounted value of the individual rewards [Mazur, J.E., 1984. J. Exp. Anal. Behav. 46, 67–77]. Three experiments reported here extend their work to sequences of unequal number of rewards and test the specific prediction that rats prefer sequences that get worse with time over sequences that improve, given that the total reward rate is the same. Choice functions from this and other two experiments that offered choice between unequal reward numbers were S-shaped and supported a signal detection-like model in which the value of each option obeys the parallel discounting model.

Preference for Different Sequences of Increasing or Decreasing Rewards

Preference for different increasing (or decreasing) sequences of rewards has been found to depend both on the magnitude of increase or decrease from step-to-step in the sequence and on the rate of change with which rewards increase or decrease. This experiment examined the effects on preference of different magnitudes and rates of change of reward. Using rewards for actual task performance, four increasing and four decreasing sequences were studied, each consisting of 24 rewards of varying magnitude. Sequences differed according to four possible models of rate change: linear, exponential, logarithmic, and step function. Preferences for given reward sequences obtained prior to extensive task and reward experience (decision utility) were not closely related to preferences after such experience (predicted utility). In the increasing reward sequences, subjects preferred the step sequence (with its single large increase at the end) prior to experience, but after experience they preferred the exponential, linear, and logarithmic sequences which entailed continuous reward-to-reward improvement throughout the sequence. In the decreasing sequences, subjects were less definite in their preferences. Prior to experience they most preferred the logarithmic sequence with its decelerating decline in magnitude of rewards, while after experience they least preferred the step function, with its huge loss at the end of the sequence.

On the Rational Pricing of the “Russian Option” for the Symmetrical Binomial Model of a $(B,S)$-Market

We present in the binomial model of Cox, Rubinstein and Ross the closed form solution for the “Russian option”, i.e., the American type option with the reward sequence $f = (f_n )_{n \geqq 0} $ given by \[ f_n (\omega ) = \beta ^n \mathop {\max }\limits_{k \leqq n} S_k (\omega ), \] where $\beta $ is some discounting factor, $0 < \beta < 1$.This option was introduced earlier by L. Sheep and A. N. Shiryaev [3], in the framework of the diffusion model of Black and Sholes.

The relation between memory and expectancy as revealed by percentage and sequence of reward investigations

There is growing agreement that to explain instrumental learning properly, one should emphasize memory as well as expectancy. I call this approachmemory-expectancy theory. Amsel's (1992) frustration theory is one variety of memory-expectancy theory. Capaldi's (1994) sequential theory is another. In this report, I examine in considerable detail the effects of percentage and sequence of reward on extinction following different levels of acquisition training. These extinction findings, taken together with certain serial learning acquisition findings, seem to support a novel version of memory-expectancy theory, one that in some respects is similar to and in some respects is different from that suggested by Amsel. First, on the basis of this analysis, we may reject two ideas: that animals remember only the prior reward event and that animals anticipate only the reward event contingent upon the current response. Second, the analysis supports three salient propositions of the present memory-expectancy approach. Memories of reward events may serve as conditioned stimuli for expectancies of reward events. On any current trial, the animal may remember each of the reward events associated with one or more prior trials. On any current trial, the animal may anticipate not only the current reward event, but also reward events contingent upon subsequent trials. Essentially, according to this model, the stimuli that elicit expectancies, as well as the expectancies themselves, may change progressively over a series of learning trials.

Frustration Theory: An Analysis of Dispositional Learning and Memory

We live in a world in which inconsistency is the rule rather than the exception and this is particularly true for rewards and frustrations. In some cases, rewards and frustrative non-rewards appear randomly for what seems to be the same behaviour; in others a sequence of rewards is suddenly followed by non-rewards, or large rewards by small rewards. The important common factor in these and other cases is frustration - how we learn about it and how we respond to it. This book provides a basis in learning theory and particularly in frustration theory, for a comprehension not only of the mechanisms controlling these dispositions, but also of their order of appearance in early development and, to an approximation at least, their neural underpinnings.

Memory dynamics and foraging strategies of honeybees

The foraging behavior of a single bee in a patch of four electronic flower dummies (feeders) was studied with the aim of analyzing the informational components in the choice process. In different experimental combinations of reward rates, color marks, odors and distances of the feeders, the behavior of the test bee was monitored by a computer in real time by several devices installed in each feeder. The test bee optimizes by partially matching its choice behavior to the reward rates of the feeders. The matching behavior differs strongly between “stay” flights (the bee chooses the feeder just visited) and “shift” flights (the bee chooses one of the three alternative feeders). The probability of stay and shift flights depends on the reward sequence and on the time interval between successive visits. Since functions describing the rising probability of stay flights with rising amounts of sucrose solution just experienced differ for the four feeders, it is concluded that bees develop feeder-specific memories. The choice profiles of shift flights between the three alternative feeders depend on the mean reward rate of the feeder last visited. Good matching is found after visits to the low-reward feeders and poor matching following departure from the high-reward feeders. These results indicate that bees use two different kinds of memories to guide their choice behavior: a transient short-term working memory that is not feeder-specific, and a feeder-specific long-term reference memory. Model calculations were carried out to test this hypothesis. The model was based on a learning rule (the difference rule) developed by Rescorla and Wagner (1972), which was extended to the two forms of memories to predict this operant behavior. The experiments show that a foraging honeybee learns the properties of a food source (its signals and rewards) so effectively that specific expectations guide the choice behavior.

Coalitions, leadership, and social norms: The power of suggestion in games

This paper examines the set of outcomes sustainable by a leader with the power to make suggestions in games. By acting as focal points, these suggestions are important even if players can communicate and form coalitions. For finite-horizon games, I show that sustainable outcomes are supported by “scapegoat” strategies, which hold a single player accountable for the actions of a group. For infinite-horizon, two-player repeated games, I show that by using an appropriate sequence of punishments and rewards, a leader can induce sufficiently patient players to play any feasible, individually rational outcome. Finally, leadership power is shown to increase if coalitions must consider the credibility of deviations in a manner similar to Coalition-Proof Nash Equilibrium.

Effects of Interrun interval and Shift of Reward Sequence on Serial Pattern Learning in Rats

In order to examine the effects of interrun intervals (IRI) on serial pattern learning, two groups of nine albino rats each were trained to achieve five consecutive runs in a straight alley and exposed to a series of reward magnitude varying number of 45mg food pellets at each run. In the first stage of the training, the first group (Group S) was presented with 14-7-3-1-0 monotonic series of food pellets with IRI of 15 to 20 sec between runs. The second group (Group L) was also presented with the same series as the first one but with longer IRI of 5 min between runs. Group S and Group L ran equally slow to 0 pellets in the monotonic series. In the second stage of the experiment, both groups received 14-3-7-1-0 nonmonotonic series with the IRI being same as the first stage. The shift of the series differentially affected the anticipation of the 0 pellets between groups. Anticipation of 0 pellet in Group S was more greatly deteriorated than that in Group L. These results support Hulse's dual theoretical approach to serial pattern learning rather than Capaldi's memory-discrimination theory.

The Asymptotic Behavior of the Reward Sequence in the Optimal Stopping of I.I.D. Random Variables

Let $X_1, X_2,\ldots$ be integrable, i.i.d. r.v.'s with common distribution function $F$ and let $\{v_n\}_{n \geq 1}$ be the sequence of optimal rewards or values in the associated optimal stopping problem, i.e., $v_n = \sup\{E(X_T): T \text{is a stopping time for} \{X_m\}_{m\geq 1} \text{and} T \leq n\}$ for $n \geq 1$. For distribution functions $F$ in the domain of attraction of one of the three classical extreme-value laws $G_I, G^\alpha_{II}$ or $G^\alpha_{III}$, it is shown that $\lim_n n(1 - F(v_n)) = 1, 1 - \alpha^{-1}$, or $1 + \alpha^{-1}$ if $F \in \mathscr{D}(G_1), F \in \mathscr{D}(G^\alpha_{II})$ and $\alpha > 1$, or $F \in \mathscr{D}(G^\alpha_{III})$ and $\alpha > 0$, respectively. From this result, the growth rate of $\{v_n\}_{n\geq 1}$ is obtained and compared to the growth rate of the expected maximum sequence. Also, the limit distribution of the optimal reward r.v.'s $\{X_{T^\ast_n}\}_{n\geq 1}$ is derived, where $\{T^\ast_n\}_{n\geq 1}$ are the optimal stopping times defined by $T^\ast_n \equiv 1$ if $n = 1$ and, for $n = 2,3,\ldots$, by $T^\ast_n = \min\{1 \leq k v_{n-k}\}$ if this set is not equal to $\varnothing$ and equal to $n$ otherwise. This tail-distribution growth rate is shown to be sufficient for any threshold sequence to be asymptotically optimal.

The influence of reward sequence on flight directionality in bees

Honeybees ( Apis mellifera L.) were trained to collect food from arrays of artificial flowers. Experiments were carried out to investigate how different sequences of rewards (sugar solution) affect the flight directionality in the next inter-flower move (i.e. the tendency to keep to the same direction as in the preceding flight). In the texts, bees could visit a row of four or five flowers. At the end of the row, bees were given a choice of direction for their next flight (i.e. forward, left, right, or backward). I investigated the effect of a reward sequence in relation to (1) the average amount of ‘nectar’ gathered in the last few flower visits, regardless of the sequence of occurrence, or to (2) the reward received in the most recent flower visits. A reward given at the choice point itself reduced flight directionality. Effects of rewards offered on preceding flowers, i.e. one, two, or three steps before the choice point, were not significant. There was, however, evidence for their additional effect. It is concluded that the influence of a reward on flight directionality vanishes rapidly as more flower visits are made.

Extensions of the multiarmed bandit problem: The discounted case

There are N independent machines. Machine i is described by a sequence {X^{i}(s), F^{i}(s)} where X^{i}(s) is the immediate reward and F^{i}(s) is the information available before i is operated for the sth lime. At each time one operates exacfiy one machine; idle machines remain frozen. The problem is to schedule the operation of the machines so as to maximize the expected total discounted sequence of rewards. An elementary proof shows that to each machine is associated an index, and the optimal policy operates the machine with the largest current index. When the machines are completely observed Markov chains, this coincides with the well-known Gittins index rule, and new algorithms are given for calculating the index. A reformulation of the bandit problem yields the tax problem, which includes, as a special case, Klimov's waiting time problem. Using the concept of superprocess, an index rule is derived for the case where new machines arrive randomly. Finally, continuous time versions of these problems are considered for both preemptive and nonpreemptive disciplines.

Approximations to Optimal Stopping Rules for Exponential Random Variables

For $X_1, X_2, \cdots$ i.i.d. with finite mean and $Y_n = \max(X_1, \cdots, X_n) - cn, c$ positive, a number of authors have considered the problem of determining an optimal stopping rule for the reward sequence $Y_n$. The optimal stopping rule can be given explicitly in this case; however, in general its use requires complete knowledge of the distribution of the $X_i$. This paper examines the problem of approximating the optimal expected reward when only partial information about the distribution is available. Specifically, if the $X_i$ are known to be exponentially distributed with unknown mean, stopping rules designed to approximate the optimal rule (which can be used only when the mean is known) are proposed. Under certain conditions the difference between the expected reward using the proposed stopping rules and the optimal expected reward vanishes as $c$ approaches zero.

An analysis of serial pattern learning by rats

When rats receive a sequence of rewards of different magnitudes for traversing a runway, they learn to “track” the sequence, showing anticipation of the forthcoming reward by appropriate running speed. There is disagreement as to whether this behavior depends on rats’ encoding and recalling a complete sequence of foregoing hedonic events or just the immediately preceding one. The present experiments showed that rats can remember more hedonic events than the most recent one. In Experiment 1, when exposed concurrently to the sequences 10-1-0 (pellets) and 0-1-10, they were faster on Run 3 of the increasing than of the decreasing sequence, a discrimination which cannot be made on the basis of the preceding (1-pellet) reward alone. Experiment 2 showed that this behavior reflects genuine anticipation of the Run 3 reward, not simultaneous contrast or other simple aftereffects of Runs 1 and 2. It is argued, however, that these results, together with related findings by Capaldi and Verry (1981), show merely that rats can recall a hedonic event other than the most recent one, not that a sequence of such events is fully recalled in order.