Fractional Order Gradient Descent Research Articles

Improved fractional-order gradient descent method based on multilayer perceptron

The fractional-order gradient descent (FOGD) method has been employed by numerous scholars in Artificial Neural Networks (ANN), with its superior performance validated both theoretically and experimentally. However, current FOGD methods only apply fractional-order differentiation to the loss function. The application of FOGD based on Autograd to hidden layers leverages the characteristics of fractional-order differentiation, significantly enhancing its flexibility. Moreover, the implementation of FOGD in the hidden layers serves as a necessary foundation for establishing a family of fractional-order deep learning optimizers, facilitating the widespread application of FOGD in deep learning. This paper proposes an improved fractional-order gradient descent (IFOGD) method based on Multilayer Perceptron (MLP). Firstly, a fractional matrix differentiation algorithm and its fractional matrix differentiation solver is proposed based on MLP, ensuring that IFOGD can be applied within the hidden layers. Subsequently, we overcome the issue of incorrect backpropagation direction caused by the absolute value symbol, ensuring that the IFOGD method does not cause divergence in the value of the loss function. Thirdly, fractional-order Autograd (FOAutograd) is proposed based on PyTorch by reconstructing Linear layer and Mean Squared Error Loss module. By combining FOAutograd with first-order adaptive deep learning optimizers, parameter matrices in each layer of ANN can be updated using fractional-order gradients. Finally, we compare and analyze the performance of IFOGD with other methods in simulation experiments and time series prediction tasks. The experimental results demonstrate that the IFOGD method exhibits performances.

An interval neural network-based Caputo fractional-order extreme learning machine applied to classification

In this study, interval-valued data is processed using an interval neural network (INN) based on a fractional-order extreme learning machine (FOELM). The traditional gradient-based algorithms of INN learn slowly. Extreme learning machine (ELM) achieves optimal weights for the output layer in a single calculation, which is faster than iterative training algorithms. However, due to the nonlinear constraints of INN, ELM cannot compute the output layer weights through the Moore–Penrose generalized inverse. Therefore, FOELM is proposed to train the INN in this paper, which ensures faster learning and significant performance. Specifically, in a single hidden layer INN, the hidden layer weights are randomly generated and fixed, and the output layer weights are trained by the fractional-order gradient descent method (FGDM). The experimental results indicate that FOELM exhibits faster convergence and superior classification performance compared to alternative algorithms.

MFFGD: An adaptive Caputo fractional-order gradient algorithm for DNN

As a primary optimization method for neural networks, gradient descent algorithm has received significant attention in the recent development of deep neural networks. However, current gradient descent algorithms still suffer from drawbacks such as an excess of hyperparameters, getting stuck in local optima, and poor generalization. This paper introduces a novel Caputo fractional-order gradient descent (MFFGD) algorithm to address these limitations. It provides fractional-order gradient derivation and error analysis for different activation functions and loss functions within the network, simplifying the computation of traditional fractional order gradients. Additionally, by introducing a memory factor to record past gradient variations, MFFGD achieves adaptive adjustment capabilities. Comparative experiments were conducted on multiple sets of datasets with different modalities, and the results, along with theoretical analysis, demonstrate the superiority of MFFGD over other optimizers.

Self-Organizing Optimization Based on Caputo’s Fractional Order Gradients

This paper analyses the condition necessary to guarantee no divergence for Caputo’s fractional order gradient descent (C-FOG) algorithm on multivariate functions. C-FOG is self-organizing, computationally efficient, simple, and understandable. It converges faster than the classical gradient-based optimization algorithms and converges to slightly different points when the order of the fractional derivative is different. The additional freedom of the order is very useful in situations where the diversity of convergence is required, and it also allows for more precise convergence. Comparative experiments on a typical poor conditioned function and adversarial sample generation frameworks demonstrate the convergence performance of C-FOG, showing that it outperforms currently popular algorithms in terms of convergence speed, and more excitingly, the diversity of convergence allows it to exhibit stronger and more stable attack capability in adversarial sample generation procedures (The code for experiments is available at: https://github.com/mulertan/self_optimizing/tree/main, accessed on 30 April 2024).

A novel gradient descent optimizer based on fractional order scheduler and its application in deep neural networks

To improve convergence speed of deep neural network trained by gradient descent methods (GDMs), this paper proposed a novel fractional order gradient descent optimizer (FOAdam) based on Caputo operator and Adam algorithm, and analyzed the impact of fractional order on convergence performance using optimization theory and simulation. Furthermore, to address the trade-off between convergence speed and precision under fixed-order conditions, a fractional order scheduler (FOS) was designed based on the Connections Cloud Model (CCM). Numerical experiments conducted on MNIST and CIFAR-10 datasets showed that FOS-FOAdam outperformed other mainstream GDMs in terms of convergence speed and precision. Finally, FOS-FOAdam was applied in recognizing and classifying image datasets of cuttings and cores at the logging site, demonstrating good prospects for practical engineering applications.

The Improved Stochastic Fractional Order Gradient Descent Algorithm

This paper mainly proposes some improved stochastic gradient descent (SGD) algorithms with a fractional order gradient for the online optimization problem. For three scenarios, including standard learning rate, adaptive gradient learning rate, and momentum learning rate, three new SGD algorithms are designed combining a fractional order gradient and it is shown that the corresponding regret functions are convergent at a sub-linear rate. Then we discuss the impact of the fractional order on the convergence and monotonicity and prove that the better performance can be obtained by adjusting the order of the fractional gradient. Finally, several practical examples are given to verify the superiority and validity of the proposed algorithm.

Fault diagnosis of rolling bearing based on BP neural network with fractional order gradient descent

The health of rolling bearing is of great importance for the normal operation of rotating machinery. The fault diagnosis process can be roughly summarized as signal processing, feature extraction, and fault classification. In this paper, a novel feature extraction and fault diagnosis method with fractional order back-propagation neural network is put forward. The new sine cosine algorithm optimized variational mode decomposition is performed on vibration signals, and the fault feature vectors are selected and built by singular value decomposition. Inspired by the fractional order calculus, a fractional order back-propagation neural network is employed to realize fault classification. The capability of the developed fault diagnosis algorithm is comprehensively evaluated via benchmark bearing data. The experimental results demonstrate that the designed method substantially extracts bearing defect features, increases classification accuracy and efficiency, and also outperforms existing algorithms.

Lateral control of intelligent vehicles using radial basis function neural networks with sliding mode control based on fractional order calculus

PurposeThis paper aimed a fractional-order sliding mode-based lateral lane-change control method that was proposed to improve the path-tracking accuracy of vehicle lateral motion.Design/methodology/approachIn this paper the vehicle presighting and kinematic models were established, and a new sliding mode control isokinetic convergence law was devised based on the fractional order calculus to make the front wheel turning angle approach the desired value quickly. On this basis, a fractional gradient descent algorithm was proposed to adjust the radial basis function (RBF) neuron parameter update rules to improve the compensation speed of the neural network.FindingsThe simulation results revealed that, compared to the traditional sliding mode control strategy, the designed controller eliminated the jitter of the sliding mode control, sped up the response of the controller, reduced the overshoot of the system parameters and facilitated accurate and fast tracking of the desired path when the vehicle changed lanes at low speeds.Originality/valueThis paper combines the idea of fractional order calculus with gradient descent algorithm, proposed a fractional-order gradient descent method applied to RBF neural network and fast adjustment the position and width of neurons.

Sensitivity of Fractional-Order Recurrent Neural Network with Encoded Physics-Informed Battery Knowledge

In the field of state estimation for the lithium-ion battery (LIB), model-based methods (white box) have been developed to explain battery mechanism and data-driven methods (black box) have been designed to learn battery statistics. Both white box methods and black box methods have drawn much attention recently. As the combination of white box and black box, physics-informed machine learning has been investigated by embedding physic laws. For LIB state estimation, this work proposes a fractional-order recurrent neural network (FORNN) encoded with physics-informed battery knowledge. Three aspects of FORNN can be improved by learning certain physics-informed knowledge. Firstly, the fractional-order state feedback is achieved by introducing a fractional-order derivative in a forward propagation process. Secondly, the fractional-order constraint is constructed by a voltage partial derivative equation (PDE) deduced from the battery fractional-order model (FOM). Thirdly, both the fractional-order gradient descent (FOGD) and fractional-order gradient descent with momentum (FOGDm) methods are proposed by introducing a fractional-order gradient in the backpropagation process. For the proposed FORNN, the sensitivity of the added fractional-order parameters are analyzed by experiments under the federal urban driving schedule (FUDS) operation conditions. The experiment results demonstrate that a certain range of every fractional-order parameter can achieve better convergence speed and higher estimation accuracy. On the basis of the sensitivity analysis, the fractional-order parameter tuning rules have been concluded and listed in the discussion part to provide useful references to the parameter tuning of the proposed algorithm.

Physics-Informed Recurrent Neural Networks with Fractional-Order Constraints for the State Estimation of Lithium-Ion Batteries

The state estimation of lithium-ion battery is the basis of an intelligent battery management system; therefore, both model-based and data-driven methods have been designed and developed for state estimation. Rather than using complex partial differential equations and the complicated parameter tuning of a model-based method, a machine learning algorithm provides a new paradigm and has been increasingly applied to cloud big-data platforms. Although promising, it is now recognized that big data for machine learning may not be consistent in terms of data quality with reliable labels. Moreover, many algorithms are still applied as a black box that may not learn battery inner information well. To enhance the algorithm generalization in realistic situations, this paper presents a fractional-order physics-informed recurrent neural network (PIRNN) for state estimation. The fractional-order characteristics from battery mechanism are embedded into the proposed algorithm by introducing fractional-order gradients in backpropagation process and fractional-order constraints into the convergence loss function. With encoded battery knowledge, the proposed fractional-order PIRNN would accelerate the convergence speed in training process and achieve improved prediction accuracies. Experiments of four cells under federal urban driving schedule operation conditions and different temperatures are conducted to illustrate the estimation effects of the proposed fractional-order PIRNN. Compared to the integer-order gradient descent method, the fractional-order gradient descent method proposed in this work can optimize network convergence and obtains regression coefficient larger than 0.995. Moreover, the experimental results indicate that the proposed algorithm can achieve 2.5% estimation accuracy with the encoding fractional-order knowledge of lithium-ion batteries.

Fractional Gradient Based Optimization for Nonlinear Separable Data

The Support Vector Machine or SVM classifier is one of the machine learning algorithms whose job is to predict data. Traditional classifier has limitations in the process of training large-scale data, tends to be slow. This study aims to increase the efficiency of the SVM classifier using a fractional gradient descent optimization algorithm, so that the speed of the data training process can be increased when using large-scale data. There are ten numerical data sets used in the simulation that are used to test the performance of the SVM classifier that has been optimized using the Caputo type fractional gradient descent algorithm. In this paper, we use the Caputo derivative formula to calculate the fractional-order gradient descent from the error function with respect to weights and obtain a deterministic convergence to increase the speed of the Caputo type fractional-order derivative convergence. The test results show that the optimized SVM classifier achieves a faster convergence time with iterations and a small error value. For further research, the optimized SVM linear classifier with fractional gradient descent is implemented on the problem of unbalanced class data.

Study on fast speed fractional order gradient descent method and its application in neural networks

This article introduces a novel fractional order gradient descent method for the quadratic loss function. Based on Riemann-Liouville definition, a more practical fractional order gradient descent method with variable initial value is proposed to ensure convergence to the actual extremum. On this basis, the random weight particle swarm optimization algorithm is introduced to select the appropriate initial value, which not only accelerates the convergence speed, but also enhances the global convergence ability of the algorithm. To avoid complicated problems of the chain rule in fractional calculus, the parameters of output layers is trained by the new designed method, while the parameters of hidden layers still use the conventional method. By selecting proper hyper-parameters, the proposed method shows faster convergence speed than others. Finally, numerical examples are given to verify that the proposed algorithm has fast convergence speed and high accuracy under a adequate large number of independent runs.

Physics-Informed Recurrent Neural Network With Fractional-Order Gradients for State-of-Charge Estimation of Lithium-Ion Battery

As a typical machine learning algorithm, neural networks (NNs) has been designed and developed for battery management system (BMS) with artificial intelligence. State of charge (SOC) estimation of lithium-ion battery (LIB) is the basis of BMS so as to widely employ NNs, and recurrent neural network (RNN) is usually selected to describe the time-series characteristics of LIB. However, RNN is a data-driven statistic black box, which cannot reveal electrochemical principle and learn inner Knowledge of LIB. This paper introduces fractionalorder gradients for RNN to improve its backpropagation process, so that network updates weights instructed by the fractionalorder characteristics of LIB. Our work provides two backpropagation patterns with fractional-order gradient descent and momentum for RNN, respectively, both resulting in a physicsinformed RNN for SOC estimation of LIB. The proposed physicsinformed RNN can conduct training in which the gradients and the loss of network is informed by the physical fractional-order laws of LIB. Experimental results under operation conditions of federal urban driving schedule (FUDS) are presented with satisfying SOC estimation accuracy. Furtherly, physics-informed RNN proposed in this paper is not limited to SOC estimation, but also other state estimation or even fault prognosis for LIB.

Forecasting Economic Growth of the Group of Seven via Fractional-Order Gradient Descent Approach

This paper establishes a model of economic growth for all the G7 countries from 1973 to 2016, in which the gross domestic product (GDP) is related to land area, arable land, population, school attendance, gross capital formation, exports of goods and services, general government, final consumer spending and broad money. The fractional-order gradient descent and integer-order gradient descent are used to estimate the model parameters to fit the GDP and forecast GDP from 2017 to 2019. The results show that the convergence rate of the fractional-order gradient descent is faster and has a better fitting accuracy and prediction effect.

Vessel Track Prediction Based on Fractional Gradient Recurrent Neural Network with Maneuvering Behavior Identification

To improve the accuracy of ship track prediction, a fractional-order gradient descent method is adopted into a recurrent neural network (RNN). The convergence of the proposed algorithm is proved. Identification of ship maneuvering behavior, atmospheric information, and oceanographic information is considered in vessel tack prediction. The ship track of Xiamen Port is predicted using the new algorithm. Error analysis is made with different factional orders and traffic busy degrees. Results show that the testing and training error differs with different fractional orders. The predicted track results can not only improve the efficiency of marine traffic management but also prevent and warn the safety accidents, so as to avoid accidents.

Low light image enhancement based on modified Retinex optimized by fractional order gradient descent with momentum RBF neural network

To dynamically adjust the edge preservation and smoothness of low-light images, this paper proposed a fractional order gradient descent with momentum radial basis function neural network (FOGDMRBF) to optimizing Retinex.Its convergence is proved. In order to speed up the convergence process, an adaptive learning rate is used to adjust the training process reasonably. The results verify the theoretical results of the proposed algorithm such as its monotonicity and convergence. The descending curve of error values by FOGDM is more smoother than gradient descent and gradient descent with momentum method. The influence of regularization parameter is analyzed and compared. Compared with Dark Channel Prior, Histogram Equalization, Homomorphic Filtering and Multiple Exposure Fusion, the halo and noise generated are significantly reduced with higher Peak Signal-to-Noise Ratio and Structural Similarity Index.

Fractional-order gradient descent with momentum for RBF neural network-based AIS trajectory restoration

Fractional-order gradient descent with momentum for RBF neural network-based AIS trajectory restoration

A quasi fractional order gradient descent method with adaptive stepsize and its application in system identification

In this paper, the fractional order gradient method (FOGM) is extended to the solution of high-dimensional function optimization problems. A quasi fractional order gradient descent method (QFOGDM) is proposed and then introduce an adaptive stepsize into QFOGDM. The theoretic analysis for convergence of QFOGDM is be done by three theorems. The numerical experiments for solving 15 unconstrained optimization benchmarks are compared to show its’ better performance. Meanwhile, the proposed algorithm is utilized to identify the parameters in the linear discrete deterministic systems and achieves a better convergence rate and accuracy.

Data classification based on fractional order gradient descent with momentum for RBF neural network

ABSTRACT The weight-updating methods have played an important role in improving the performance of neural networks. To ameliorate the oscillating phenomenon in training radial basis function (RBF) neural network, a fractional order gradient descent with momentum method for updating the weights of RBF neural network (FOGDM-RBF) is proposed for data classification. Its convergence is proved. In order to speed up the convergence process, an adaptive learning rate is used to adjust the training process. The Iris data set and MNIST data set are used to test the proposed algorithm. The results verify the theoretical results of the proposed algorithm such as its monotonicity and convergence. Some non-parametric statistical tests such as Friedman test and Quade test are taken for the comparison of the proposed algorithm with other algorithms. The influence of fractional order, learning rate and batch size is analysed and compared. Error analysis shows that the algorithm can effectively accelerate the convergence speed of gradient descent method and improve its performance with high accuracy and validity.

Weighted Internal Model Control-Proportional Integral Derivative Control Scheme Via Fractional Gradient Descent Algorithm

Abstract An adaptive controller design technique based on internal model control (IMC) scheme is proposed in this paper. Multiple IMC controllers having different values of filter time constants and exhibiting widely different performance are combined via weight update rule. The weight update rule, formulated via convex combination of integral and fractional order gradient descent algorithms, assigns time varying weights to individual candidate controllers to obtain an enhanced performance over the individual candidate controllers. The beauty of the proposed technique is that it employs the simplicity of one degree-of-freedom (1DOF) IMC structure to achieve an improved performance over existing 2DOF control schemes. The efficacy of the proposed technique is demonstrated via three illustrative examples and via experimental validation on the hardware setup of dc servosystem. An extensive comparative analysis in terms of simulation plots and performance indices offers a testimony to the effectiveness of the proposed scheme.