Decreasing Step Size Research Articles

Measurement-based optimal routing on overlay architectures for unicast sessions

We propose a measurement-based routing algorithm to load-balance intradomain traffic along multiple paths for multiple unicast sources. Multiple paths are established using overlay nodes. The distributed algorithm is derived from simultaneous perturbation stochastic approximation (SPSA) and does not assume that the gradient of an analytical cost function is known. Instead, we rely on (potentially) noisy estimates from local measurements. Using the analytical model presented in the paper, we first show the almost sure convergence of the algorithm to the optimal solution under a decreasing step size policy (as with a standard SPSA model). Motivated by practical concerns, we next consider the constant step size case, for which we establish weak convergence. We provide simulation results to demonstrate the advantages of our proposed algorithm under various network scenarios, and also present a comparative study with MATE (an existing optimal routing algorithm).

Solving a non-convex combined travel forecasting model by the method of successive averages with constant step sizes

The method of successive averages is often used in iterative algorithms for solving various mathematical problems, and travel forecasting models in particular. In each iteration of these algorithms the current solution is averaged with an alternative solution generated by the algorithm. If the problem has an equivalent convex optimization formulation with a computable objective function, a line search can be used to determine the weight applied to the alternative solution, referred to as the step size. In the absence of a convex optimization formulation, the typical approach is to set the step size according to a predetermined sequence that is decreasing towards zero, such as 1/ k where k is the iteration index. In this paper, we examine the use of a constant step size in the method of successive averages. A theoretical derivation shows that if the alternative solution generated by the algorithm is a linear function of the current solution, then using a constant step size is advantageous and substantially superior to using a sequence of decreasing step sizes. We conjecture that similar results may be expected in nonlinear differentiable problems as well. Numerical results are presented for a travel forecasting model that combines user-equilibrium route choice with an origin-destination-mode (ODM) choice model. The proposed algorithm is based on origin-based assignment, in conjunction with successive averaging of constant step sizes for the differentiable ODM model. The results show that a properly chosen constant step size leads to excellent convergence for both convex and non-convex models. A general strategy for choosing step sizes without a-priori knowledge is presented as well.

The Validity of the Additive Noise Model for Uniform Scalar Quantizers

A uniform scalar quantizer with small step size, large support, and midpoint reconstruction levels is frequently modeled as adding orthogonal noise to the quantizer input. This paper rigorously demonstrates the asymptotic validity of this model when the input probability density function (pdf) is continuous and satisfies several other mild conditions. Specifically, as step size decreases, the correlation between input and quantization error becomes negligible relative to the mean-squared error (MSE). The model is even valid when the input density is discontinuous at the origin, but discontinuities elsewhere can prevent the correlation from being negligible. Though this invalidates the additive model, an asymptotic formula for the correlation is found in terms of the step size and the heights and positions of the discontinuities. For a finite support input density, such as uniform, it is shown that the support of the uniform quantizer can be matched to that of the density in ways that make the correlation approach a variety of limits. The derivations in this paper are based on an analysis of the asymptotic convergence of cell centroids to cell midpoints. This convergence is fast enough that the centroids and midpoints induce the same asymptotic MSE, but not fast enough to induce the same correlations.

Algorithms and networks for accelerated convergence of adaptive LDA

We introduce and discuss new accelerated algorithms for linear discriminant analysis (LDA) in unimodal multiclass Gaussian data. These algorithms use a variable step size, optimally computed in each iteration using (i) the steepest descent, (ii) conjugate direction, and (iii) Newton–Raphson methods in order to accelerate the convergence of the algorithm. Current adaptive methods based on the gradient descent optimization technique use a fixed or a monotonically decreasing step size in each iteration, which results in a slow convergence rate. Furthermore, the convergence of these algorithms depends on appropriate choices of the step sizes. The new algorithms have the advantage of automatic optimal selection of the step size using the current data samples. Based on the new adaptive algorithms, we present self-organizing neural networks for adaptive computation of Σ - 1 / 2 and use them in cascaded form with a PCA network for LDA. Experimental results demonstrate fast convergence and robustness of the new algorithms and justify their advantages for on-line pattern recognition applications with stationary and non-stationary multidimensional input data.

Fast adaptive algorithms and networks for class-separability features

In this article, we introduce accelerated algorithms for linear discriminant analysis (LDA) and feature extraction from unimodal multiclass Gaussian data. Current adaptive methods based on the gradient descent optimization technique use a fixed or a monotonically decreasing step size in each iteration, which results in a slow convergence rate. Here, we use a variable step size, optimally computed in each iteration using the steepest descent method, in order to accelerate the convergence of the algorithm. Based on the new adaptive algorithm, we present a self-organizing neural network for adaptive computation of the square root of the inverse covariance matrix ( Σ −1/2) and use it (i) in a network for optimal feature extraction from Gaussian data and (ii) in cascaded form with a principal component analysis network for LDA. Experimental results demonstrate fast convergence and high stability of the algorithm and justify its advantages for on-line pattern recognition applications with stationary and non-stationary input data.

NLMS algorithm with decreasing step size for adaptive IIR filters

In this paper, we modify the GLMS algorithm to become the proposed LMS-SAS algorithm, which can more effectively converge to the global minimum of the mean-square output error (MSE) objective function. We also derive the infinite impulse response-normalized least-squares algorithm (IIR-NLMS), whose behavior is similar to the LMS-SAS algorithm. The GLMS algorithm achieves its global search capability by appending a random perturbing noise to the LMS algorithm. Similarly, we suggest that such perturbing noise is to be multiplied by its MSE objective function in the proposed LMS-SAS algorithm. For the NLMS algorithm, we use the gradient estimation error, which exists naturally in the adaptive process, to act as perturbing noise. We have shown, theoretically and experimentally, that the LMS-SAS and NLMS algorithm do converge to the global minimum.

Block adaptive filters with deterministic reference inputs for event-related signals: BLMS and BRLS

Adaptive estimation of the linear coefficient vector in truncated expansions is considered for the purpose of modeling noisy, recurrent signals. Two different criteria are studied for block-wise processing of the signal: the mean square error (MSE) and the least squares (LS) error. The block LMS (BLMS) algorithm, being the solution of the steepest descent strategy for minimizing the MSE, is shown to be steady-state unbiased and with a lower variance than the LMS algorithm. It is demonstrated that BLMS is equivalent to an exponential averager in the subspace spanned by the truncated set of basis functions. The block recursive least squares (BRLS) solution is shown to be equivalent to the BLMS algorithm with a decreasing step size. The BRLS is unbiased at any occurrence number of the signal and has the same steady-state variance as the BLMS but with a lower variance at the transient stage. The estimation methods can be interpreted in terms of linear, time-variant filtering. The performance of the methods is studied on an ECG signal, and the results show that the performance of the block algorithms is superior to that of the LMS algorithm. In addition, measurements with clinical interest are found to be more robustly estimated in noisy signals.

Training neural networks with additive noise in the desired signal

A new global optimization strategy for training adaptive systems such as neural networks and adaptive filters [finite or infinite impulse response (FIR or IIR)] is proposed in this paper. Instead of adding random noise to the weights as proposed in the past, additive random noise is injected directly into the desired signal. Experimental results show that this procedure also speeds up greatly the backpropagation algorithm. The method is very easy to implement in practice, preserving the backpropagation algorithm and requiring a single random generator with a monotonically decreasing step size per output channel. Hence, this is an ideal strategy to speed up supervised learning, and avoid local minima entrapment when the noise variance is appropriately scheduled.

Sharp convergence rates of stochastic approximation for degenerate roots

Sharp convergence rates of stochastic approximation algorithms are given for the case where the derivative of the unknown regression function at the sought-for root is zero. The convergence rates obtained are sharp for the general step size used in the algorithms in contrast to the previous work where they are not sharp for slowly decreasing step sizes; all possible limit points are found for the case where the first matrix coefficient in the expansion of the regression function is normal; and the estimation upper bound is shown to be achieved for the multi-dimensional case in contrast to the previous work where only the one-dimensional result is proved.

Richardson Plots for Fractal Dimension Characterization of Bound Aries of Digitized Grains and Particles

Abstract Fractal dimension is useful for characterizing objects with tortuous boundaries, providing an additional measurement to characterize shape beyond the commonly used area, perimeter, caliper diameter and their combinations. The fractal dimension of the boundary describes the increase of the measured value of the perimeter of an object resulting from increasing measurement resolution, i.e., decreasing step size. We desire to characterize and classify shapes such as soot particles, as shown in the TEM micrograph in Fig. 1a, and crystalline grains of metal alloys, as shown in the SEM micrograph of Raney nickel in Fig. 4. This investigation involves evaluation of the Minkowski-sausage or dilation technique (Fig. 1) using the histogram of the Euclidean distance map of the perimeter of the object. The distance map gives bands of various widths that follow the perimeter - greater widths lose detail. Perimeter length is sausage (band) area divided by sausage width, w.

Convergence and Error Bounds for Passive Stochastic Algorithms Using Vanishing Step Size

This work is concerned with passive stochastic approximation (PSA) algorithms having vanishing or decreasing step size and window width. Unlike the traditional stochastic approximation methods, the passive stochastic approximation algorithms utilize passive strategies. Under the framework of PSA, not only the measurement noise is unobservable, but also the “state” {xn} is a randomly generated sequence. In our formulation, both the observation noise and the randomly generated {xn} are correlated random processes. Under rather general conditions, w.p.1. convergence of the algorithms is established. Then upper bounds on estimation errors are obtained. It is shown that the bounds depend on the smoothness of the function under consideration in an essential way, which reveals another distinct feature of the passive algorithms.

Convergence of the delayed normalized LMS algorithm with decreasing step size

In several practical applications of the LMS algorithm, including certain VLSI implementations, the coefficient adaptation can be performed only after some fixed delay. The resulting algorithm is known as the delayed LMS (DLMS) algorithm in the literature. Previous published analyses of this algorithm are based on mean and moment convergence under the independence assumption between successive input vectors. These analyses are interesting and give valuable insights into the convergence properties but, from a practical viewpoint, they do not guarantee the correct performance of the particular realization with which the user must live. We consider a normalized version of this algorithm with a decreasing step size /spl mu/(n) and prove the almost sure convergence of the nonhomogeneous algorithm, assuming a mixing input condition and the satisfaction of a certain law of large numbers.

Passive stochastic approximation with constant step size and window width

Motivated by the desire to solve a steady-state estimation and detection problem in chemical engineering, and inspired by the recent progress in passive stochastic approximation, recursive algorithms combining stochastic approximation and kernel estimation are studied and developed in this work. The underlying problem can be stated as finding the roots f(x)=0 provided only noisy measurements y/sub n/=f(x/sub n/,/spl xi//sub n/) are available, where Ef(x, /spl xi//sub n/)=f(x). The main difficulty lies in that unlike the traditional approach in stochastic approximation, the sequence {x/sub n/} is generated randomly and cannot be adjusted in accordance with our wish. Similar to those used in the decreasing step size algorithms, another sequence {z/sub n/} is generated to approximate the roots of f~(x)=0. Some of the features of the algorithms include: constant step size and constant window width and correlated random processes. Under fairly general conditions, it is proven that a weak convergence result holds for an interpolated sequence of the iterates. Error bounds are obtained and a local limit theorem is also derived. The algorithm is then applied to an estimation problem in chemical engineering. Simulation has shown promising results.

Analyzing the (1, λ) Evolution Strategy via Stochastic Approximation Methods

The main objective of this paper is to analyze the (1, λ) evolution strategy by use of stochastic approximation methods. Both constant and decreasing step size algorithms are studied. Convergence and estimation error bounds for the (1, λ) evolution strategy are developed. First the algorithm is converted to a recursively defined scheme of stochastic approximation type. Then the analysis is carried out by using the analytic tools from stochastic approximation. In lieu of examining the discrete iterates, suitably scaled sequences are defined. These interpolated sequences are then studied in detail. It is shown that the limits of the sequences have natural connections to certain continuous time dynamical systems.

Asymptotic Properties of a Class of Stochastic Recursive Algorithms

The subject matter discussed in this paper is on a class of stochastic approximation/optimization algorithms. The underlying problems can be stated as: Find the solution of f(x) = 0 provided only noise measurements are available and the design points {xn} is a sequence generated passively and randomly. Our main effort is on developing asymptotic properties of the algorithms. Decreasing step size algorithms will be studied in detail. In addition to proving the convergence of the algorithms, we derive the upper bounds on the estimation errors.

Convergence analysis of an adaptive filter equipped with the sign-sign algorithm

Convergence with probability 1 of the sign-sign algorithm with a decreasing step size is proven when it is used in adaptive plant identification. The plant input and plant noise are assumed stationary and M-dependent. The assumptions do not pose any upper bound on the memory, M, of the plant input and plant noise, and hence, they allow arbitrarily strong correlation in these signals. A main interest of the paper is that it provides a condition of persistent excitation of the algorithm. Examples satisfying this condition are provided. >

Establishing connections between evolutionary algorithms and stochastic approximation

This work is our first attempt in establishing the connections between evolutionary computation algorithms and stochastic approximation procedures. By treating evolutionary algorithms as recursive stochastic procedures, we study both constant gain and decreasing step size algorithms. We formulate the problem in a rather general form, and supply the sufficient conditions for convergence (both with probability one, and in the weak sense). Among other things, our approach reveals the natural connection of the discrete iterations and the continuous dynamics (ordinary differential equations, and/or stochastic differential equations). We hope that this attempt will open up a new horizon for further research and lead to in depth understanding of the underlying algorithms.

Convergence of an adaptive linear estimation algorithm

In this work we prove the almost sure convergence of an adaptive linear estimator governed by a stochastic gradient algorithm with decreasing step size in the presence of correlated observations. Two complementary contributions are added to the famous 1977 Ljung theorem. First we drop the condition of nondivergence of the algorithm assumed by Ljung. While that condition can be ensured by adding a barrier, the convergence of the suitably bounded algorithm itself is not established even on the basis of Ljung theorem. Here, the barrier problem is overcome by proving that it is not necessary for the convergence. Our second contribution is to generalize the model describing the correlated observations. No state space model is used and no linear relationship between the observations and the signal to be estimated needs to be assumed. Instead we use a decreasing covariance model that agrees with a very wide class of practical applications.

Variable-gradient method for adaptive equaliser

A simple technique for the implementation of an adaptive equaliser is a gradient-seeking method. Two critical parameters associated with adaptive equalisation are the rate of convergence and the steady-state mean-square error attainable. When a fixed gradient is used in the gradient-search procedure for an adaptive equaliser, rapid convergence and a small steady-state mean-square error are two conflicting requirements. Use of a variable gradient parameter at each iteration provides a good compromise. A method for iteratively computing the gradient parameter is introduced, so that the random walk in the gradient-search procedure is executed, on the average, with decreasing step size as the iterative algorithm converges.

Developing and Analyzing Newton – C’otes Quadrature Formulae for Approximating Definite Integrals- AC++ Approach

In this paper, different Newton – C’otes quadrature formulae for the approximation of definite integrals and their error analysis are derived. The order of convergences of the methods is also derived and of these Newton – C’otes quadrature formulae, the Simpson’s 1/3 rule have been shown to have high order of convergence. Since the functionality of these numerical integration methods is practical only if we can use computer programs and applications to produce approximate solutions with acceptable errors within short period, C++ programs for the selected methods are written. These programs are used on the comparison of the Newton – C’otes quadrature formulae and the result obtained based on the inputs and outputs of the programs for different integrands. The results of these programs show that the convergence of the methods highly depends on the number of iterations. The results of different numerical examples show that for high accuracy of the trapezoidal rule computational effort is higher and round off errors with large number of iterations limit the accuracy. The results show that the Simpson’s 1/3 rule produces much more accurate solution than other methods even within small number of iterations. This shows that the error for Simpson’s rule 1/3 converges to zero faster than the error for the trapezoidal rule as the step size decreases. It is finally observed that Simpson’s 1/3 rule is much faster than the Trapezoidal and the Simpson’s 3/8 rules according to the results of the C++ programs.