Gradient-based Adaptive Algorithms Research Articles

Chance constrained dynamic optimization approach for single machine scheduling involving flexible maintenance, production, and uncertainty

Chance constraints are suitable for industrial process modeling under uncertain conditions, where constraints cannot be strictly satisfied or do not need to be fully satisfied. In this paper, a single machine scheduling problem involving flexible maintenance, production, and uncertainty is modeled as a chance constrained dynamic optimization problem (CCDOP). A novel method is proposed for transforming the CCDOP into an equivalent deterministic dynamic optimization problem (DOP) with fixed state jump times. Furthermore, by using the idea of l1 penalty function and a smooth approximation technique, the resulting deterministic DOP becomes a smoothing penalty problem, which is a non-convex nonlinear parameter optimization problem (NNPOP) with simple bounds on the variables. To solve the NNPOP, a gradient-based stochastic search algorithm (GSSA) is developed based on a gradient-based adaptive search algorithm (GASA) and a novel stochastic search algorithm (NSSA). The convergence analysis result shows that the GSSA is a globally convergent algorithm. Finally, two numerical examples are used to illustrate the effectiveness of the proposed method. Numerical results show that the GSSA has excellent convergence behavior with robust computation feature, providing better results compared with the other typical methods.

A fractional taylor series-Based least mean square algorithm, and its application to power signal estimation

The definition of fractional derivative by Caputo and Riemann inspired the researchers to develop new adaptive algorithms having better convergence properties in comparison with integer-gradient based adaptive algorithms. As reported in many studies, the existing fractional gradient-based adaptive algorithms lack justification for using fractional derivative in addition to integer-gradient, may become inconsistent in the event of negative weights, and may yield more or less same performance as compared to the conventional integer-derivative-based algorithms by appropriate selection of step-size parameter. Accordingly, this paper presents a novel fractional adaptive algorithm based on Fractional Taylor Series. Unlike (most of) the existing fractional-derivative-based algorithms, the proposed algorithm only involves fractional-derivative and ensures convergence of mean square error (MSE) provided the step-size is chosen appropriately. Simulation results are presented in order to depict a scenario where exploitation of fractional-derivative in the weight-update equation yields better convergence as compared to LMS algorithm in the context of power signal parameters estimation.

Mutual Information Based Learning Rate Decay for Stochastic Gradient Descent Training of Deep Neural Networks.

This paper demonstrates a novel approach to training deep neural networks using a Mutual Information (MI)-driven, decaying Learning Rate (LR), Stochastic Gradient Descent (SGD) algorithm. MI between the output of the neural network and true outcomes is used to adaptively set the LR for the network, in every epoch of the training cycle. This idea is extended to layer-wise setting of LR, as MI naturally provides a layer-wise performance metric. A LR range test determining the operating LR range is also proposed. Experiments compared this approach with popular alternatives such as gradient-based adaptive LR algorithms like Adam, RMSprop, and LARS. Competitive to better accuracy outcomes obtained in competitive to better time, demonstrate the feasibility of the metric and approach.

LMSK: a robust higher‐order gradient‐based adaptive algorithm

It has been shown that, in intensely noisy environments, adaptive algorithms based on higher-order statistics can enjoy better performance, as compared with the well known second-order least-mean-square (LMS) algorithm. By contrast, this advantage diminishes for low signal-to-noise ratio (SNR) levels, where the LMS algorithm outperforms. One remedy is to employ the LMS algorithm in conjunction with a higher-order adaptation algorithm, in a mixed mode. Least-mean kurtosis (LMK) is a higher-order algorithm that has been shown to be advantageous to use if the noise distribution is Gaussian or super-Gaussian. In this study, the authors propose the LMS/kurtosis algorithm, a stochastic gradient-based adaptive algorithm that is a combination of the LMS and the LMK algorithms. Simulation results demonstrate the privilege of the proposed algorithm, in comparison with its counterparts, for a wide range of noise distributions and SNR levels. This improvement is achieved in spite of a negligible increase in computational complexity. An analytical model is also derived for the mean weight as well as the weight-error covariance matrix, from which the mean-square-error behaviour of the algorithm can be predicted. Simulation results show the high accuracy of the derived model in different conditions.

A $ q$-Gaussian Maximum Correntropy Adaptive Filtering Algorithm for Robust Sparse Recovery in Impulsive Noise

This letter proposes a robust formulation for sparse signal reconstruction from compressed measurements corrupted by impulsive noise, which exploits the $\boldsymbol {q}$ -Gaussian generalized correntropy $\boldsymbol {(1 as the loss function for the residual error and utilizes a $\ell _{0}$ -norm penalty term for sparsity inducing. To solve this formulation efficiently, we develop a gradient-based adaptive filtering algorithm which incorporates a zero-attracting regularization term into the framework of adaptive filtering. This new proposed algorithm blending the advantages of adaptive filtering and $\boldsymbol {q}$ -Gaussian generalized correntropy can obtain accurate reconstruction and satisfactory robustness with a proper shape parameter $\boldsymbol {q}$ . Numerical experiments on both synthetic sparse signals and natural images are conducted to illustrate the superior recovery performance of the proposed algorithm to the state-of-the-art robust sparse signal reconstruction algorithms.

Robust Gradient-Based Adaptive Control of Nonlinearly Parametrized Plants

Most of the contributions in adaptive control literature assume that the system dynamics is linearly parametrizable, and a certainty equivalence principle is exploited to guarantee global stability and asymptotic convergence of tracking error to zero. Although linear-in-the-parameters (LIP) assumption is reasonable for a large class of dynamics, there exists a considerable number of real world systems, involving complex dynamics, where nonlinear parametrizations are inevitable. Previous research has shown that classical gradient-based adaptive designs with the certainty equivalence principle do not perform satisfactorily for nonlinear-in-the-parameter (NLIP) systems, and may, in fact, cause instability in many situations. This letter is an attempt toward addressing this issue by a novel non-certainty equivalence adaptive control design, where the classical gradient-based adaptive algorithm is used to tackle the LIP component of the NLIP dynamics, while a robust compensator, appended to the controller, accounts for the linearization error. The designed controller ensures global uniformly ultimately bounded stability of the error dynamics. Simulation results on a model of biochemical process, involving NLIP dynamics, are provided to validate the theoretical development.

Compressive Sensing Signal Reconstruction Using L0-Norm Normalized Least Mean Fourth Algorithms

Stochastic gradient-based adaptive algorithm has recently attracted considerable attention as one of the best candidates for solving compressive sensing (CS) problems due to its two obvious advantages: low complexity and robust performance. In this paper, in order to further improve the reconstruction accuracy for CS problems under Gaussian background noise, two novel sparse fourth-order error criterion adaptive algorithms, i.e., the $${\ell }_{{0}}$$ -norm normalized least mean fourth ( $${\ell }_{{0}}$$ -NLMF) algorithm and the $${\ell }_{{0}}$$ -norm exponentially forgetting window NLMF ( $${\ell }_{{0}}$$ -EFWNLMF) algorithm, are proposed. In addition, to extend the obtained results to non-Gaussian noise environment, the variants of the above two algorithms, i.e., the sign $${\ell }_{{0}}$$ -NLMF ( $${\ell }_{{0}}$$ -SNLMF) algorithm and the sign $${\ell }_{{0}}$$ -EFWNLMF ( $${\ell }_{{0}}$$ -EFWSNLMF) algorithm, are presented which can effectively mitigate certain impulsive noises. Numerical simulations are also given to demonstrate the evident performance improvement and extensive stability of the proposed algorithms.

Steady-State Tracking Analysis of Adaptive Filter With Maximum Correntropy Criterion

This letter studies the tracking performance of a stochastic gradient-based adaptive algorithm, namely the maximum correntropy criterion algorithm, where a random walk is used to model the non-stationarity. In our analysis, we use the energy conservation argument to derive expressions for the steady-state excess mean square error (EMSE). We consider two different cases for measurement of noise distribution including the Gaussian noise and general non-Gaussian noise. For the Gaussian case, we derive a fixed-point equation that can be solved numerically to find steady-state EMSE value. For the general non-Gaussian case, we derive an approximate closed-form expression for EMSE. For both cases, unlike the stationary environment, the EMSE curves are not increasing functions of step size parameter. We use this observation to find the optimum step size learning parameter for general non-Gaussian case. The validity of the theoretical results are justified via simulation results.

Single-channel acoustic echo cancellation in noise based on gradient-based adaptive filtering

In this paper, a two-stage scheme is proposed to deal with the difficult problem of acoustic echo cancellation (AEC) in single-channel scenario in the presence of noise. In order to overcome the major challenge of getting a separate reference signal in adaptive filter-based AEC problem, the delayed version of the echo and noise suppressed signal is proposed to use as reference. A modified objective function is thereby derived for a gradient-based adaptive filter algorithm, and proof of its convergence to the optimum Wiener-Hopf solution is established. The output of the AEC block is fed to an acoustic noise cancellation (ANC) block where a spectral subtraction-based algorithm with an adaptive spectral floor estimation is employed. In order to obtain fast but smooth convergence with maximum possible echo and noise suppression, a set of updating constraints is proposed based on various speech characteristics (e.g., energy and correlation) of reference and current frames considering whether they are voiced, unvoiced, or pause. Extensive experimentation is carried out on several echo and noise corrupted natural utterances taken from the TIMIT database, and it is found that the proposed scheme can significantly reduce the effect of both echo and noise in terms of objective and subjective quality measures.

A Normalized Least Mean Squares Algorithm With a Step-Size Scaler Against Impulsive Measurement Noise

This brief introduces the concept of a step-size scaler by investigating and modifying the tanh cost function for adaptive filtering with impulsive measurement noise. The step-size scaler instantly scales down the step size of gradient-based adaptive algorithms whenever impulsive measurement noise appears, which eliminates a possibility of updating weight vector estimates based on wrong information due to impulsive noise. The most attractive feature of the step-size scaler is that this is easily applicable to various gradient-based adaptive algorithms. Several representative gradient-based adaptive algorithms are performed without or with the step-size scaler in impulsive-noise environments, which shows the improvement of robustness against impulsive noise.

Weighted non‐linear criterion‐based adaptive generalised eigendecomposition

Generalised eigendecomposition problem for a symmetric matrix pencil is reinterpreted as an unconstrained minimisation problem with a weighted non-linear criterion. The analytical results show that the proposed criterion has a unique global minimum which corresponds to the principal generalised eigenvectors, thus guaranteeing the global convergence via iterative methods to search the minimum. A gradient-based adaptive algorithm and a fixed point iteration-based adaptive algorithm are derived for the generalised eigendecomposition, which both work in parallel and avoid the error propagation effect of sequential-type algorithms. By applying the stochastic approximation theory, the global convergence of the proposed adaptive algorithm is proved. The performance of the proposed method is evaluated by simulations in terms of convergence rate, estimation accuracy as well as tracking capability.

An Adaptive Nonlinear Filter for System Identification

The primary difficulty in the identification of Hammerstein nonlinear systems (a static memoryless nonlinear system in series with a dynamic linear system) is that the output of the nonlinear system (input to the linear system) is unknown. By employing the theory of affine projection, we propose a gradient-based adaptive Hammerstein algorithm with variable step-size which estimates the Hammerstein nonlinear system parameters. The adaptive Hammerstein nonlinear system parameter estimation algorithm proposed is accomplished without linearizing the systems nonlinearity. To reduce the effects of eigenvalue spread as a result of the Hammerstein system nonlinearity, a new criterion that provides a measure of how close the Hammerstein filter is to optimum performance was used to update the step-size. Experimental results are presented to validate our proposed variable step-size adaptive Hammerstein algorithm given a real life system and a hypothetical case.

Dually Optimal Neuronal Layers: Lobe Component Analysis

Development imposes great challenges. Internal ldquocorticalrdquorepresentations must be autonomously generated from interactive experiences. The eventual quality of these developed representations is of course important. Additionally, learning must be as fast as possible-to quickly derive better representation from limited experiences. Those who achieve both of these will have competitive advantages. We present a cortex-inspired theory called lobe component analysis (LCA) guided by the aforementioned dual criteria. A lobe component represents a high concentration of probability density of the neuronal input space. We explain how lobe components can achieve a dual-spatiotemporal (ldquobestrdquo and ldquofastestrdquo)-optimality, through mathematical analysis, in which we describe how lobe components plasticity can be temporally scheduled to take into account the <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">history</i> of observations in the best possible way. This contrasts with using only the last observation in gradient-based adaptive learning algorithms. Since they are based on two cell-centered mechanisms-Hebbian learning and lateral inhibition-lobe components develop <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">in-place</i> , meaning every networked neuron is individually responsible for the learning of its signal-processing characteristics within its connected network environment. There is no need for a separate learning network. We argue that in-place learning algorithms will be crucial for real-world large-size developmental applications due to their simplicity, low computational complexity, and generality. Our experimental results show that the learning speed of the LCA algorithm is drastically faster than other Hebbian-based updating methods and independent component analysis algorithms, thanks to its dual optimality, and it does not need to use any second- or higher order statistics. We also introduce the new principle of fast learning from stable representation.

Analysis of the Desired-Response Influence on the Convergence of Gradient-Based Adaptive Algorithms

Although the convergence behavior of gradient-based adaptive algorithms, such as steepest descent and leas mean square (LMS), has been extensively studied, the influence of the desired response on the transient convergence has generally received little attention. However, empirical results show that this signal can have a great impact on the learning curve. In this paper we analyze the influence of the desired response on the transient convergence by making a novel interpretation, from the viewpoint of the desired response, of previous convergence analyses of SD and LMS algorithms. We show that, without prior knowledge that can be used to wisely select the initial weight vector, initial convergence is fast whenever there is high similarity between input and desired response whereas, on the contrary, when there is low similarity between these two signals, convergence is slow from the beginning.

A Globally Convergent Stochastic Pairwise Conjugate Gradient-Based Algorithm for Adaptive Filtering

It is well known that, for the deterministic optimization problem, the method of conjugate gradient has superior convergence rates compared to ordinary gradient methods. For quadratic problems, the conjugate gradient method has finite termination property which makes it one of the most favorable iterative methods. However, the fast convergence and the finite termination property can easily break down when the function to be optimized is noisy since conjugacy among search directions can no longer be maintained over the course of the iterations. In this letter, a conjugation procedure is applied in an adaptive filtering algorithm, where, instead of producing a set of conjugate search directions, only pairwise conjugation of gradients is produced during each system update. Simulations show that the algorithm provides superior convergence compared to the stochastic gradient descent algorithm, and comparable to existing conjugate gradient-based adaptive filtering algorithms, but at a lower computational cost.

Wavelet feature domain adaptive noise reduction using learning algorithm for text-independent speaker recognition

In this paper, a type of thresholding method is developed for adaptive noise reduction. Here, we propose a new type thresholding method. Unlike the standard thresholding functions, the new thresholding functions are infinitely differentiable. Gradient-based adaptive learning algorithms are presented to seek the optimal solution for noise reduction. Furthermore, the learning algorithm can be used for any speaker data derived from discrete wavelet transform. It is demonstrated that 94% correct classification rates can be achieved by the use of the first 32 variation features in TALUNG database.

On-Line Adaptation of Grid-Based Look-up Tables Using a Fast Linear Regression Technique

Advanced control systems require accurate process models, while processes are often both nonlinear and time variant. After introducing the identification of nonlinear processes with grid-based look-up tables, a new learning algorithm for on-line adaptation of look-up tables is proposed. Using a linear regression approach, this new adaptation algorithm considerably reduces the convergence time in relation to conventional gradient-based adaptation algorithms. An application example and experimental results are shown for the learning feedforward control of the ignition angle of a spark ignition engine.

Thresholding neural network for adaptive noise reduction

In the paper, a type of thresholding neural network (TNN) is developed for adaptive noise reduction. New types of soft and hard thresholding functions are created to serve as the activation function of the TNN. Unlike the standard thresholding functions, the new thresholding functions are infinitely differentiable. By using the new thresholding functions, some gradient-based learning algorithms become possible or more effective. The optimal solution of the TNN in a mean square error (MSE) sense is discussed. It is proved that there is at most one optimal solution for the soft-thresholding TNN. General optimal performances of both soft and hard thresholding TNNs are analyzed and compared to the linear noise reduction method. Gradient-based adaptive learning algorithms are presented to seek the optimal solution for noise reduction. The algorithms include supervised and unsupervised batch learning as well as supervised and unsupervised stochastic learning. It is indicated that the TNN with the stochastic learning algorithms can be used as a novel nonlinear adaptive filter. It is proved that the stochastic learning algorithm is convergent in certain statistical sense in ideal conditions. Numerical results show that the TNN is very effective in finding the optimal solutions of thresholding methods in an MSE sense and usually outperforms other noise reduction methods. Especially, it is shown that the TNN-based nonlinear adaptive filtering outperforms the conventional linear adaptive filtering in both optimal solution and learning performance.

A new forced LMS-based adaptive algorithm utilizing the principle of potential energy

In this paper, we propose a new gradient-based adaptive algorithm, in which the conventional LMS update recursion is modified by introducing a damping factor, which emulates the force acting on a free-moving object in a gravitational field. The contribution of the damping factor being introduced is controlled by an exponential function, which eliminates its effect whenever the squared error signal is less than a given threshold. Furthermore, we show via stability analysis that the control parameter can assume both positive and negative values. Implementing a negative control parameter transforms our algorithm to a time-varying Leaky-type LMS algorithm, which exhibits bias-free performance, unlike that for the conventional Leaky-LMS algorithm. With positive contribution, the algorithm exhibits improved convergence speeds with a smoothing property in stationary, nonstationary power and correlated noisy environments. Several simulation examples are persented to verify the validity of the new algorithm compared with the LMS, including echo cancellation in telephone networks and noise cancellation in transient and PCG signals.

On gradient adaptation with unit-norm constraints

In this correspondence, we describe gradient-based adaptive algorithms within parameter spaces that are specified by ||w||=1, where ||/spl middot/|| is any vector norm. We provide several algorithm forms and relate them to true gradient procedures via their geometric structures. We also give algorithms that mitigate an inherent numerical instability for L/sub 2/-norm-constrained optimization tasks. Simulations showing the performance of the techniques for independent component analysis are provided.