Abstract
A batch variable learning rate gradient descent algorithm is proposed to efficiently train a neuro-fuzzy network of zero-order Takagi-Sugeno inference systems. By using the advantages of regularization, the smoothing L <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1/2</sub> regularization is utilized to find more appropriate sparse network. Combining the second-order information of the smoothing error function, a variable learning rate is chosen along the steep descent direction, which avoids line search procedure and may reduce the cost of computation. In order to appropriately adjust the Lipschitz constant of the smoothing error function in the learning rate, a new scheme is proposed by introducing a hyper-parameter. Also the article applies the modified secant equation for estimating the Lipschitz constant, which makes the algorithm greatly reduce the oscillating phenomenon and improve the robustness. Under appropriate assumptions, a convergent result of the proposed algorithm is also given. Simulation results for two identification and classification problems show that the proposed algorithm has better numerical performance and promotes the sparsity capability of the network, compared with the common batch gradient descent algorithm and a variable learning rate gradient-based algorithm.
Highlights
Regularization is one of the important approaches for the overfitting problem
For neuro-fuzzy systems, in order to reduce the adverse effects of high non-linearity and ill-condition of the error function and improve the numerical performance of the algorithm and generalization capacity of the network, we shall use the second-order information from the iterations to define a learning rate without line search, and further propose a batch gradient-based algorithm with the smoothing L1/2 regularization
The advantages of the proposed algorithm and the main contribution of the paper are as follows: (1) We have proposed a new scheme for adjusting the Lipschitz constant to update the learning rate by introducing a hyper-parameter
Summary
Regularization is one of the important approaches for the overfitting problem. A regularization term is generally introduced on the basis of minimizing the empirical risk to limit the model capabilities so that they do not excessively minimize empirical risk. For neuro-fuzzy systems, in order to reduce the adverse effects of high non-linearity and ill-condition of the error function and improve the numerical performance of the algorithm and generalization capacity of the network, we shall use the second-order information from the iterations to define a learning rate without line search, and further propose a batch gradient-based algorithm with the smoothing L1/2 regularization. BATCH VARIABLE LEARNING RATE GRADIENT NEURO-FUZZY LEARNING ALGORITHM WITH SMOOTHING L1/2 REGULARIZATION In order to train the neuro-fuzzy network, it usually needs to solve an unconstrained problem on the error function. A hyper-parameter c is introduced to appropriately adjust the estimate value of the Lipschitz constant in proposed algorithm, and we will firstly propose the variable learning rate gradient algorithm with smoothing L1/2 regularization, denotes as VLRGSL1/2.
Sign-in/Register to view highlights & summary 
Published Version (
Free)
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call