Optimal Learning Rate Research Articles

Distributed learning with multi-penalty regularization

In this paper, we study distributed learning with multi-penalty regularization based on a divide-and-conquer approach. Using Neumann expansion and a second order decomposition on difference of operator inverses approach, we derive optimal learning rates for distributed multi-penalty regularization in expectation. As a byproduct, we also deduce optimal learning rates for multi-penalty regularization, which was not given in the literature. These results are applied to the distributed manifold regularization and optimal learning rates are given.

Learning theory of distributed spectral algorithms

Spectral algorithms have been widely used and studied in learning theory and inverse problems. This paper is concerned with distributed spectral algorithms, for handling big data, based on a divide-and-conquer approach. We present a learning theory for these distributed kernel-based learning algorithms in a regression framework including nice error bounds and optimal minimax learning rates achieved by means of a novel integral operator approach and a second order decomposition of inverse operators. Our quantitative estimates are given in terms of regularity of the regression function, effective dimension of the reproducing kernel Hilbert space, and qualification of the filter function of the spectral algorithm. They do not need any eigenfunction or noise conditions and are better than the existing results even for the classical family of spectral algorithms.

Distributed Kernel-Based Gradient Descent Algorithms

We study the generalization ability of distributed learning equipped with a divide-and-conquer approach and gradient descent algorithm in a reproducing kernel Hilbert space (RKHS). Using special spectral features of the gradient descent algorithms and a novel integral operator approach, we provide optimal learning rates of distributed gradient descent algorithms in probability and partly conquer the saturation phenomenon in the literature in the sense that the maximum number of local machines to guarantee the optimal learning rates does not vary if the regularity of the regression function goes beyond a certain quantity. We also find that additional unlabeled data can help relax the restriction on the number of local machines in distributed learning.

Learning Rates for Classification with Gaussian Kernels.

This letter aims at refined error analysis for binary classification using support vector machine (SVM) with gaussian kernel and convex loss. Our first result shows that for some loss functions, such as the truncated quadratic loss and quadratic loss, SVM with gaussian kernel can reach the almost optimal learning rate provided the regression function is smooth. Our second result shows that for a large number of loss functions, under some Tsybakov noise assumption, if the regression function is infinitely smooth, then SVM with gaussian kernel can achieve the learning rate of order [Formula: see text], where [Formula: see text] is the number of samples.

Optimal Learning Rates for Kernel Partial Least Squares

We study two learning algorithms generated by kernel partial least squares (KPLS) and kernel minimal residual (KMR) methods. In these algorithms, regularization against overfitting is obtained by early stopping, which makes stopping rules crucial to their learning capabilities. We propose a stopping rule for determining the number of iterations based on cross-validation, without assuming a priori knowledge of the underlying probability measure, and show that optimal learning rates can be achieved. Our novel analysis consists of a nice bound for the number of iterations in a priori knowledge-based stopping rule for KMR and a stepping stone from KMR to KPLS. Technical tools include a recently developed integral operator approach based on a second order decomposition of inverse operators and an orthogonal polynomial argument.

An Interval-Valued Fuzzy Cerebellar Model Neural Network Based on Intuitionistic Fuzzy Sets

An interval-valued fuzzy cerebellar model neural network (IV-FCMNN) is proposed for the identification and control of uncertain systems. It is a more general model that uses the framework of a cerebellar model neural network (CMNN) and Atanassov intuitionistic fuzzy sets (AIFSs), so that the mathematical representation of a fuzzy event is more complete and the fuzzy neural network is more general. In some special cases, this neural network can be reduced to an interval-valued fuzzy neural network (IV-FNN), a fuzzy neural network (FNN), a fuzzy cerebellar model neural network (FCMNN) or a CMNN. Since the interval-type input data are used to realize this algorithm, the IV-FCMNN copes better with uncertainty and allows greater freedom of design. Therefore, the ability to learn, the approximation precision and the fuzzy semantic description of this network are much better than those of other models. In the proposed IV-FCMNN, a training algorithm that uses a gradient descent method is proposed to adjust the parameters and convergence is proved using the Lyapunov stability theorem. The variable learning rates are analyzed and the optimal learning rates are also determined. For demonstrating the effectiveness of the proposed IV-FCMNN, three types of applications, including multiple functions approximation, multi-dimensional classification and nonlinear dynamic system feedback control, are performed and the comparison with other models are also provided.

RETRACTED ARTICLE: Nonlinear backstepping control design of LSM drive system using adaptive modified recurrent Laguerre orthogonal polynomial neural network

The good control performance of the permanent magnet linear synchronous motor (LSM) drive system is very difficult to achieve using linear controller because of uncertainty effects, such as ending-fictitious force. A backstepping approach is proposed to control the motion of the LSM drive system. With the proposed backstepping control system, the mover position of the LSM drive achieves good transient control performance and robustness. Although favorable tracking responses can be obtained by the backstepping control system, the chattering in the control effort is critical because of the large control gain. Because there are many nonlinear and time-varying uncertainties in the LSM drive systems, the nonlinear backsteping control system, which an adaptive modified recurrent Laguerre orthogonal polynomial neural network (NN) is used to estimate uncertainty, is thus proposed to reduce the chattering in the control effort and thereby enhance the robustness of the LSM drive system. In addition, the on-line parameter training methodology of the modified recurrent Laguerre orthogonal polynomial NN is based on the Lyapunov stability theorem. Furthermore, two optimal learning rates of the modified recurrent Laguerre orthogonal polynomial NN are derived to accelerate parameter convergence. Finally, comparison of the experimental results of the present study demonstrates the high control performance of the proposed control scheme.

PSO-Self-Organizing Interval Type-2 Fuzzy Neural Network for Antilock Braking Systems

Nowadays, the antilock braking system (ABS) is the standard in all modern cars. The function of ABS is to optimize the maximize wheel traction by preventing wheel lockup during braking, so it can help the drivers to maintain steering maneuverability. In this study, a self-organizing interval type-2 fuzzy neural network (SOT2FNN) control system is designed for antilock braking systems. This control system comprises a main controller and a robust compensation controller; the SOT2FNN as the main controller is used to mimic an ideal controller, and the robust compensation controller is developed to eliminate the approximation error between the main controller and the ideal controller. To guarantee system stability, adaptive laws for adjusting the parameters of SOT2FNN based on the gradient descent method are proposed. However, in control design, the learning rates of adaptive law are very important and they significantly affect control performance. The particle swarm optimization method is therefore applied to find the optimal learning rates for the weights in reduction layer and also for the means, the variances of the Gaussian functions in the input membership functions. Finally, the numerical simulations of ABS response in different road conditions are provided to illustrate the effectiveness of the proposed approach.

Greedy Criterion in Orthogonal Greedy Learning

Orthogonal greedy learning (OGL) is a stepwise learning scheme that starts with selecting a new atom from a specified dictionary via the steepest gradient descent (SGD) and then builds the estimator through orthogonal projection. In this paper, we found that SGD is not the unique greedy criterion and introduced a new greedy criterion, called as " -greedy threshold" for learning. Based on this new greedy criterion, we derived a straightforward termination rule for OGL. Our theoretical study shows that the new learning scheme can achieve the existing (almost) optimal learning rate of OGL. Numerical experiments are also provided to support that this new scheme can achieve almost optimal generalization performance while requiring less computation than OGL.

Divide and conquer local average regression

The divide and conquer strategy, which breaks a massive data set into a se- ries of manageable data blocks, and then combines the independent results of data blocks to obtain a final decision, has been recognized as a state-of-the-art method to overcome challenges of massive data analysis. In this paper, we merge the divide and conquer strategy with local average regression methods to infer the regressive relationship of input-output pairs from a massive data set. After theoretically analyzing the pros and cons, we find that although the divide and conquer local average regression can reach the optimal learning rate, the restric- tion to the number of data blocks is a bit strong, which makes it only feasible for small number of data blocks. We then propose two variants to lessen (or remove) this restriction. Our results show that these variants can achieve the optimal learning rate with much milder restriction (or without such restriction). Extensive experimental studies are carried out to verify our theoretical assertions.

A Six-Phase CRIM Driving CVT using Blend Modified Recurrent Gegenbauer OPNN Control

Because the nonlinear and time-varying characteristics of continuously variable transmission (CVT) systems driven by means of a six-phase copper rotor induction motor (CRIM) are unconscious, the control performance obtained for classical linear controllers is disappointing, when compared to more complex, nonlinear control methods. A blend modified recurrent Gegenbauer orthogonal polynomial neural network (OPNN) control system which has the online learning capability to come back to a nonlinear time-varying system, was complied to overcome difficulty in the design of a linear controller for six-phase CRIM driving CVT systems with lumped nonlinear load disturbances. The blend modified recurrent Gegenbauer OPNN control system can carry out examiner control, modified recurrent Gegenbauer OPNN control, and reimbursed control. Additionally, the adaptation law of the online parameters in the modified recurrent Gegenbauer OPNN is established on the Lyapunov stability theorem. The use of an amended artificial bee colony (ABC) optimization technique brought about two optimal learning rates for the parameters, which helped reform convergence. Finally, a comparison of the experimental results of the present study with those of previous studies demonstrates the high control performance of the proposed control scheme.

RETRACTED: Novel application of continuously variable transmission system using composite recurrent Laguerre orthogonal polynomials modified PSO NN control system

Because the V-belt continuously variable transmission system spurred by permanent magnet (PM) synchronous motor has much unknown nonlinear and time-varying characteristics, the better control performance design for the linear control design is a time consuming procedure. In order to overcome difficulties for design of the linear controllers, the composite recurrent Laguerre orthogonal polynomials modified particle swarm optimization (PSO) neural network (NN) control system which has online learning capability to come back to the nonlinear and time-varying of system, is developed for controlling PM synchronous motor servo-driven V-belt continuously variable transmission system with the lumped nonlinear load disturbances. The composite recurrent Laguerre orthogonal polynomials NN control system consists of an inspector control, a recurrent Laguerre orthogonal polynomials NN control with adaptation law and a recouped control with estimation law. Moreover, the adaptation law of online parameters in the recurrent Laguerre orthogonal polynomials NN is originated from Lyapunov stability theorem. Additionally, two optimal learning rates of the parameters by means of modified PSO are posed in order to achieve better convergence. At last, comparative studies shown by experimental results are illustrated to demonstrate the control performance of the proposed control scheme.

Deep learning with adaptive learning rate using laplacian score

An attempt has been made to improve the performance of Deep Learning with Multilayer Perceptron (MLP). Tuning the learning rate or finding an optimum learning rate in MLP is a major challenge. Depending on the value of the learning rate, classification accuracy can vary drastically. This issue has been taken as a challenge in this paper. In this paper, a new approach has been proposed to combine adaptive learning rate in conjunction with the concept of Laplacian score for varying the weights. Learning rate is taken as a function of parameter which itself is updated on the basis of error gradient by forming mini-batches. Laplacian score of the neuron isfurther used for updating the incoming weights. This removes the bottleneck involved in finding the optimum value for the learning rate in Deep Learning by using MLP. It is observed on benchmark datasets that this approach leads to increase in classification accuracy as compared to the existing benchmark levels achieved by the well known methods of deep learning.

Learning rates for the risk of kernel-based quantile regression estimators in additive models

Additive models play an important role in semiparametric statistics. This paper gives learning rates for regularized kernel-based methods for additive models. These learning rates compare favorably in particular in high dimensions to recent results on optimal learning rates for purely nonparametric regularized kernel-based quantile regression using the Gaussian radial basis function kernel, provided the assumption of an additive model is valid. Additionally, a concrete example is presented to show that a Gaussian function depending only on one variable lies in a reproducing kernel Hilbert space generated by an additive Gaussian kernel, but does not belong to the reproducing kernel Hilbert space generated by the multivariate Gaussian kernel of the same variance.

A Backstepping Control of LSM Drive Systems Using Adaptive Modified Recurrent Laguerre OPNNUO

The good control performance of permanent magnet linear synchronous motor (LSM) drive systems is difficult to achieve using linear controllers because of uncertainty effects, such as fictitious forces. A backstepping control system using adaptive modified recurrent Laguerre orthogonal polynomial neural network uncertainty observer (OPNNUO) is proposed to increase the robustness of LSM drive systems. First, a field-oriented mechanism is applied to formulate a dynamic equation for an LSM drive system. Second, a backstepping approach is proposed to control the motion of the LSM drive system. With the proposed backstepping control system, the mover position of the LSM drive achieves good transient control performance and robustness. As the LSM drive system is prone to nonlinear and time-varying uncertainties, an adaptive modified recurrent Laguerre OPNNUO is proposed to estimate lumped uncertainties and thereby enhance the robustness of the LSM drive system. The on-line parameter training methodology of the modified recurrent Laguerre OPNN is based on the Lyapunov stability theorem. Furthermore, two optimal learning rates of the modified recurrent Laguerre OPNN are derived to accelerate parameter convergence. Finally, the effectiveness of the proposed control system is verified by experimental results.

Modelling and control of six‐phase induction motor servo‐driven continuously variable transmission system using blend modified recurrent Gegenbauer orthogonal polynomial neural network control system and amended artificial bee colony optimization

AbstractBecause the nonlinear and time‐varying characteristics of the continuously variable transmission system operated using a six‐phase copper rotor induction motor are unknown, improving the control performance of the linear control design is time‐consuming. To capture the nonlinear and dynamic behaviour of the six‐phase copper rotor induction motor servo‐driven continuously variable transmission system, a blend modified recurrent Gegenbauer orthogonal polynomial neural network (NN) control system, which has the online learning capability to return to the nonlinear time‐varying system, was developed. The blend modified recurrent Gegenbauer orthogonal polynomial NN control system can perform overseer control, modified recurrent Gegenbauer orthogonal polynomial NN control, and recompensed control. Moreover, the adaptation law of online parameters in the modified recurrent Gegenbauer orthogonal polynomial NN is based on the Lyapunov stability theorem. The use of amended artificial bee colony optimization yielded two optimal learning rates for the parameters, which helped improve convergence. Finally, comparison of the experimental results of the present study with those of previous studies demonstrated the high control performance of the proposed control scheme. Copyright © 2016 John Wiley & Sons, Ltd.

Blend modified recurrent Gegenbauer orthogonal polynomial neural network control for six-phase copper rotor induction motor servo-driven continuously variable transmission system using amended artificial bee colony optimization

Because the non-linear and time-varying characteristics of the continuously variable transmission (CVT) system driven by using a six-phase copper rotor induction motor (IM) are unknown, improving the control performance of the linear control design is time consuming. To overcome difficulties in the design of a linear controller for the six-phase copper rotor IM servo-driven CVT system with lumped non-linear load disturbances, a blend modified recurrent Gegenbauer orthogonal polynomial neural network (NN) control system, which has the online learning capability to return to the non-linear time-varying system, was developed. The blend modified recurrent Gegenbauer orthogonal polynomial NN control system can perform overseer control, modified recurrent Gegenbauer orthogonal polynomial NN control and recompensed control. Moreover, the adaptation law of online parameters in the modified recurrent Gegenbauer orthogonal polynomial NN is based on the Lyapunov stability theorem. The use of amended artificial bee colony optimization (ABCO) yielded two optimal learning rates for the parameters, which helped improve convergence. Finally, comparison of the experimental results of the present study with those of previous studies demonstrated the high control performance of the proposed control scheme.

RETRACTED: Design of a composite recurrent Laguerre orthogonal polynomial neural network control system with ameliorated particle swarm optimization for a continuously variable transmission system

Because the nonlinear and time-varying characteristics of the V-belt continuously variable transmission system driven by a permanent magnet synchronous motor (PMSM) are unknown, improving the control performance of the linear control design is time-consuming. To overcome difficulties in the design of a linear controller for the PMSM servo-driven V-belt continuously variable transmission system with lumped nonlinear load disturbances, a composite recurrent Laguerre orthogonal polynomial neural network (NN) control system with ameliorated particle swarm optimization (PSO), which has the online learning capability to respond to the nonlinear time-varying system, was developed. The composite recurrent Laguerre orthogonal polynomial NN control system can perform inspector control, recurrent Laguerre orthogonal polynomial NN control which involves an adaptation law, and recouped control which involves an estimation law. Moreover, the adaptation law of online parameters in the recurrent Laguerre orthogonal polynomial NN is based on the Lyapunov stability theorem. The use of ameliorated particle swarm optimization yielded two optimal learning rates for the parameters, which helped improve convergence. Finally, comparison of the experimental results of the present study with those of previous studies demonstrated the high control performance of the proposed control scheme.

Optimal learning rates for localized SVMs

One of the limiting factors of using support vector machines (SVMs) in large scale applications are their super-linear computational requirements in terms of the number of training samples. To addr...

RETRACTED ARTICLE: Hybrid recurrent Laguerre-orthogonal-polynomials neural network control with modified particle swarm optimization application for V-belt continuously variable transmission system

A V-belt continuously variable transmission system driven by a permanent magnet synchronous motor has much unknown nonlinear and time-varying characteristics. In order to capture the system's nonlinear and dynamic behavior, a hybrid recurrent Laguerre-orthogonal-polynomials neural network (NN) control system with modified particle swarm optimization (PSO) is proposed for achieving online better learning capacity and faster convergence to enhance system robustness. The hybrid recurrent Laguerre-orthogonal-polynomials NN control system can perform inspected control, recurrent Laguerre-orthogonal-polynomials NN control, which involves an adaptive law, and recouped control, which involves an estimated law. Moreover, the adaptive law of online parameters in the recurrent Laguerre-orthogonal-polynomials NN is derived by means of Lyapunov stability theorem. Furthermore, two optimal learning rates of the online parameters in the recurrent Laguerre-orthogonal-polynomials NN by means of modified PSO are applied to achieve online better learning capacity and faster convergence. Finally, to show the effectiveness of the proposed control scheme, comparative studies are demonstrated by experimental results.