Perturbation Analysis Estimators Research Articles

Strong points of weak convergence: a study using RPA gradient estimation for automatic learning

This paper analyzes the behavior of adaptive control schemes for automatic learning. Estimates of the sensitivities are used in a gradient-based stochastic approximation procedure, in order to drive the process along the steepest descent trajectory in search for the optimum. The learning rates are kept constant for adaptability. For such procedures, convergence can be established in a weak sense. A model problem of a flexible machine is presented, for which the control parameter is a probability vector. We propose a new sensitivity estimator, generalizing the phantom rare perturbation analysis (RPA) estimator to multi-valued decisions. From the basic properties of the estimators, we build several updating rules based on the weak convergence theory to ensure asymptotic optimality. We illustrate the predicted theoretical behavior with computer simulations. Finally, we present the comparison between the behavior of our proposed scheme with a regenerative one for which we can establish strong convergence. Our results show that weak convergence yields a dramatic improvement in the rate of convergence, in addition to the capability of adaptation, or tracking.

Sample Path Analysis for Continuous Tandem Production Lines

We consider a tandem production line where the product flowing through the line is continuous fluid. We call such a line a continuous tandem (CT) line. We derive a recursive representation for the departure time of products in CT lines. Next, we apply sample path analysis techniques to prove that the departure time is convex (in the sense of strong stochastic convexity) in the flow rates of the machines. We then prove that the infinitesimal perturbation analysis (IPA) estimate, for the gradient of throughput with respect to machine flow rates, is strongly consistent.

Budget-Dependent Convergence Rate of Stochastic Approximation

Convergencerate results are derived for a stochastic optimization problem where a performance measure is minimized with respect to a vector parameter t. Assuming that a gradient estimator is available and that both the bias and the variance of the estimator are (known) functions of the budget devoted to its computation, the gradient estimator is employed in conjunction with a stochastic approximation (SA) algorithm. Our interest is to figure out how to allocate the total available computational budget to the successive SA iterations. The effort is devoted to solving the asymptotic version of this problem by finding the convergence rate of SA toward the optimizer, first as a function of the number of iterations and then as a function of the total computational effort. As a result the optimal rate of increase of the computational budget per iteration can be found. Explicit expressions for the case where the computational budget devoted to an iteration is a polynomial in the iteration number, and where the bias and variance of the gradient estimator are polynomials of the computational budget, are derived. Applications include the optimization of steady-state simulation models with likelihood ratio, perturbation analysis, or finite-difference gradient estimators; optimization of infinite-horizon models with discounting; optimization of functions of several expectations; and so on. Several examples are discussed. Our results readily generalize to general root-finding problems.

Smoothed Perturbation Analysis for Stationary Single-ServerQueues with Multiple Customer Classes

Recently, Konstantopoulos and Zazanis (1992, 1994) and Bremaud and Lasgouttes (1993) derive the infinitesimal perturbation analysis (IPA) estimators for the stationary and ergodic G/G/1/∞ queue using Palm calculus, where neither regenerative structure nor convex property are required and the strong consistency is ensured by ergodic theorem. This work has been motivated by them and derives the smoothed perturbation analysis (SPA) estimator on the stationary and ergodic framework. The problem here is how to treat the ’catastrophic jumps‘ on the sample path of the steady state and this is solved cleverly by using the Palm calculus. We deal with multi-class queues in this paper but our key formula is expected to be useful to any systems to which the SPA is applicable.

On Perturbation Propagation for Smoothed Perturbation Analysis

A critical component in the implementation of smoothed perturbation analysis (SPA) estimators is the propagation of conditional perturbations. Left- and right-hand second derivative estimators for a single-server queue are used to illustrate that one often has to be careful when it comes to perturbation propagation. In this particular example, the propagation can be carried out for the right-hand estimator using the original sample path, but not for the left-hand estimator. However, for this example, the difficulty can be overcome by simplifying the original left-hand derivative estimator so that no propagation is needed.

Comparison of gradient estimation techniques for queues with non-identical servers

Gradient estimation techniques for stochastic discrete-event simulation have been a major topic of research over the past decade. In this paper, we apply two of the techniques—perturbation analysis and the likelihood ratio method—to a single-queue system with non-identical multiple servers. We derive estimates for derivatives of mean steady-state system time with respect to parameters of the underlying timing distributions. In terms of perturbation analysis, we consider both an infinitesimal perturbation analysis estimator, which is biased for this problem, and two smoothed perturbation analysis estimators, one unbiased but not very practical and one approximate but easily implementable. For two servers, we provide an analytical proof of unbiasedness in steady state for the Markovian case. For the likelihood ratio method, we apply the regenerative likelihood ratio estimator. We provide simulation results for both Markovian and non-Markovian examples, and compare the performance of the various estimators. We conclude that no one method performs universally well, and provide recommendations as to when one is likely to be preferred to the others.

Addendum to "Extensions and generalizations of smoothed perturbation analysis in a generalized semi-Markov process framework"

In their original article (ibid., vol. 37, no. 10, p. 1483-1500, 1992), the authors derived smoothed perturbation analysis (SPA) estimators for a class of quite general discrete-event systems in a generalized semi-Markov process framework, and proved the unbiasedness of the estimators. A key assumption used in the proofs required that the distributions have bounded hazard rate functions. Here, we remove this assumption and in addition provide simplified proofs.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Stochastic gradient-based time-cost tradeoffs in PERT networks using simulation

Infinitesimal perturbation analysis and score function gradient estimators are developed for PERT (Program Evaluation and Review Technique) networks and analyzed for potential use for prescriptive project management. In particular, a stochastic version of the familiar deterministic “project crashing” problem is considered. A heuristic using the gradient estimators is developed and shown to give close to locally optimal performance relatively quickly. An attempt is made to characterize the amount of variability in networks that would warrant the use of this heuristic by comparing its performance experimentally to that of the standard linear programming approach using only the task means (ignoring variability).

Smoothed perturbation analysis derivative estimation for Markov chains

Using the technique of smoothed perturbation analysis, we consider steady-state performance measures for continuous-time Markov chains and derive a derivative estimator that can be estimated from a single sample path of the chain under consideration. The estimator is applicable to multi-class queueing networks, for which previous infinitesimal perturbation analysis estimators failed. A simple multi-class queueing network is used to illustrate the application of the estimator.

Sensitivity analysis for stationary and ergodic queues: additional results

Perturbation analysis estimators for expectations of possibly discontinuous functions of the time-stationary workload were derived in [2]. The expressions obtained may, however, not be valid if the customer-stationary distribution of the workload has atoms (at points other than zero). This was pointed out by Brémaud and Lasgouttes in [1]. In this note we clearly state the additional condition required for the validity of the expressions in [2]. We furthermore show how our approximation scheme can also be used to obtain the correct expressions for the right and left derivatives given in [1].

Sensitivity analysis for stationary and ergodic queues: additional results

Perturbation analysis estimators for expectations of possibly discontinuous functions of the time-stationary workload were derived in [2]. The expressions obtained may, however, not be valid if the customer-stationary distribution of the workload has atoms (at points other than zero). This was pointed out by Brémaud and Lasgouttes in [1]. In this note we clearly state the additional condition required for the validity of the expressions in [2]. We furthermore show how our approximation scheme can also be used to obtain the correct expressions for the right and left derivatives given in [1].

Uniformization based sensitivity estimation for a class of discrete-event systems

Infinitestimal Perturbation Analysis (IPA) estimators are based on particular couplings of parameteric families of discrete event systems where “small” changes in the parameter value, typically, cause “small” changes in the timing of events and, for finite horizons, the sequence of states visisted remains the same. We consider another coupling approach based on the uniformization procedure and a simple generalization of it. In our case any “small” change in the parameter value causes a change in the state of the system; our parameterization of trajectories keeps them highly synchronized, hence the effect of such changes can be estimated, sometimes efficiently. In this framework, we define three tupes of performance sensitivity estimators for a broad class of performance measures and with respect to a range of parameter values. Performance measures on finite deterministic horizons are considered and it is shown that they are unbiased under mild conditions. We show that for some systems the derivative estimators can be calculated from a nominal sample path of the system.

Optimization of Queues Using an Infinitesimal Perturbation Analysis-Based Stochastic Algorithm with General Update Times

Convergence (with probability one) of a stochastic optimization algorithm for a single server queue is proved. The parameter to be optimized is updated using an infinitesimal perturbation analysis estimate of the gradient of the performance measure, and the updates are performed at general times. First, an algorithm in which the parameter is updated before each customer begins service is considered. Then it is shown how this analysis can be extended to an algorithm that updates at certain stopping times. The analysis suggests that the sample path behavior of the algorithm is similar to that of an algorithm that updates at the start of every busy period.

Smoothed perturbation analysis algorithms for estimating the derivatives of occupancy-related functions in serial queueing networks

The authors present smoothed perturbation analysis (SPA) estimators for the derivative of a number of occupancy-related functions in serial queueing networks with finite buffer spaces. The functions are the average number of customers at a network as seen by an arrival, the probability that a customer is blocked at a particular queue, and the probability that a customer leaves a queue empty. In all three cases, the variable is a parameter of the distribution of service times at one of the queues. The derivative estimators considered are very simple and flexible, and they easily lend themselves to analysis of unbiasedness. Unlike most of the established SPA estimators, the present ones are not based on the computation of hazard rates. >

Sensitivity analysis for stationary and ergodic queues

Starting with some mild assumptions on the parametrization of the service process, perturbation analysis (PA) estimates are obtained for stationary and ergodic single-server queues. Besides relaxing the stochastic assumptions, our approach solves some problems associated with the traditional regenerative approach taken in most of the previous work in this area. First, it avoids problems caused by perturbations interfering with the regenerative structure of the system. Second, given that the major interest is in steady-state performance measures, it examines directly the stationary version of the system, instead of considering performance measures expressed as Cesaro limits. Finally, it provides new estimators for general (possibly discontinuous) functions of the workload and other steady-state quantities.

Sensitivity analysis for stationary and ergodic queues

Starting with some mild assumptions on the parametrization of the service process, perturbation analysis (PA) estimates are obtained for stationary and ergodic single-server queues. Besides relaxing the stochastic assumptions, our approach solves some problems associated with the traditional regenerative approach taken in most of the previous work in this area. First, it avoids problems caused by perturbations interfering with the regenerative structure of the system. Second, given that the major interest is in steady-state performance measures, it examines directly the stationary version of the system, instead of considering performance measures expressed as Cesaro limits. Finally, it provides new estimators for general (possibly discontinuous) functions of the workload and other steady-state quantities.

Scheduling arrivals to a stochastic service mechanism

We consider the problem of scheduling the arrivals of a fixed number of customers to a stochastic service mechanism to minimize an expected cost associated with operating the system. We consider the special case of exponentially distributed service times and the problems in general associated with obtaining exact analytic solutions. For general service time distributions we obtain approximate numerical solutions using a stochastic version of gradient search employing Infinitesimal Perturbation Analysis estimates of the objective function gradient obtained via simulation.

Stationary waiting time derivatives

We investigate the stability of waiting-time derivatives when inputs to a queueing system-service times and interarrival times-depend on a parameter. We give conditions under which the sequence of waiting-time derivatives admits a stationary distribution, and under which the derivatives converge to the stationary regime from all initial conditions. Further hypotheses ensure that the expectation of a stationary waiting-time derivative is, in fact, the derivative of the expected stationary waiting time. This validates the use of simulation-based infinitesimal perturbation analysis estimates with a variety of queueing processes.

Stationary likelihood ratios and smoothed perturbation analysis gradient estimates for the routing problem

We present stationary and regenerative form estimates for the gradients of the cycle variables with respect to a thinning parameter in the arrival process of G/G/1 queueing systems. Our estimates belong to the category of the likelihood ratio method (LRM) and smoothed perturbation analysis (SPA) estimates. The results are useful in adaptive routing design.

Convexity of sample path performance and strong consistency of infinitesimal perturbation analysis estimates

A strong consistency of infinitesimal perturbation analysis (IPA) estimates for a class of stochastic processes satisfying certain convex properties is established. The approach is based on convex analysis of sample paths. Applications of the results to tandem and cyclic queueing systems are discussed. Possible extensions to more general systems are also illustrated. >