Order Gradient Descent Research Articles

Self-Organizing Optimization Based on Caputo’s Fractional Order Gradients

This paper analyses the condition necessary to guarantee no divergence for Caputo’s fractional order gradient descent (C-FOG) algorithm on multivariate functions. C-FOG is self-organizing, computationally efficient, simple, and understandable. It converges faster than the classical gradient-based optimization algorithms and converges to slightly different points when the order of the fractional derivative is different. The additional freedom of the order is very useful in situations where the diversity of convergence is required, and it also allows for more precise convergence. Comparative experiments on a typical poor conditioned function and adversarial sample generation frameworks demonstrate the convergence performance of C-FOG, showing that it outperforms currently popular algorithms in terms of convergence speed, and more excitingly, the diversity of convergence allows it to exhibit stronger and more stable attack capability in adversarial sample generation procedures (The code for experiments is available at: https://github.com/mulertan/self_optimizing/tree/main, accessed on 30 April 2024).

A novel gradient descent optimizer based on fractional order scheduler and its application in deep neural networks

To improve convergence speed of deep neural network trained by gradient descent methods (GDMs), this paper proposed a novel fractional order gradient descent optimizer (FOAdam) based on Caputo operator and Adam algorithm, and analyzed the impact of fractional order on convergence performance using optimization theory and simulation. Furthermore, to address the trade-off between convergence speed and precision under fixed-order conditions, a fractional order scheduler (FOS) was designed based on the Connections Cloud Model (CCM). Numerical experiments conducted on MNIST and CIFAR-10 datasets showed that FOS-FOAdam outperformed other mainstream GDMs in terms of convergence speed and precision. Finally, FOS-FOAdam was applied in recognizing and classifying image datasets of cuttings and cores at the logging site, demonstrating good prospects for practical engineering applications.

The Improved Stochastic Fractional Order Gradient Descent Algorithm

This paper mainly proposes some improved stochastic gradient descent (SGD) algorithms with a fractional order gradient for the online optimization problem. For three scenarios, including standard learning rate, adaptive gradient learning rate, and momentum learning rate, three new SGD algorithms are designed combining a fractional order gradient and it is shown that the corresponding regret functions are convergent at a sub-linear rate. Then we discuss the impact of the fractional order on the convergence and monotonicity and prove that the better performance can be obtained by adjusting the order of the fractional gradient. Finally, several practical examples are given to verify the superiority and validity of the proposed algorithm.

Fault diagnosis of rolling bearing based on BP neural network with fractional order gradient descent

The health of rolling bearing is of great importance for the normal operation of rotating machinery. The fault diagnosis process can be roughly summarized as signal processing, feature extraction, and fault classification. In this paper, a novel feature extraction and fault diagnosis method with fractional order back-propagation neural network is put forward. The new sine cosine algorithm optimized variational mode decomposition is performed on vibration signals, and the fault feature vectors are selected and built by singular value decomposition. Inspired by the fractional order calculus, a fractional order back-propagation neural network is employed to realize fault classification. The capability of the developed fault diagnosis algorithm is comprehensively evaluated via benchmark bearing data. The experimental results demonstrate that the designed method substantially extracts bearing defect features, increases classification accuracy and efficiency, and also outperforms existing algorithms.

Study on fast speed fractional order gradient descent method and its application in neural networks

This article introduces a novel fractional order gradient descent method for the quadratic loss function. Based on Riemann-Liouville definition, a more practical fractional order gradient descent method with variable initial value is proposed to ensure convergence to the actual extremum. On this basis, the random weight particle swarm optimization algorithm is introduced to select the appropriate initial value, which not only accelerates the convergence speed, but also enhances the global convergence ability of the algorithm. To avoid complicated problems of the chain rule in fractional calculus, the parameters of output layers is trained by the new designed method, while the parameters of hidden layers still use the conventional method. By selecting proper hyper-parameters, the proposed method shows faster convergence speed than others. Finally, numerical examples are given to verify that the proposed algorithm has fast convergence speed and high accuracy under a adequate large number of independent runs.

Hierarchical Quasi-Fractional Gradient Descent Method for Parameter Estimation of Nonlinear ARX Systems Using Key Term Separation Principle

Recently, a quasi-fractional order gradient descent (QFGD) algorithm was proposed and successfully applied to solve system identification problem. The QFGD suffers from the overparameterization problem and results in estimating the redundant parameters instead of identifying only the actual parameters of the system. This study develops a novel hierarchical QFDS (HQFGD) algorithm by introducing the concepts of hierarchical identification principle and key term separation idea. The proposed HQFGD is effectively applied to solve the parameter estimation problem of input nonlinear autoregressive with exogeneous noise (INARX) system. A detailed investigation about the performance of HQFGD is conducted under different disturbance conditions considering different fractional orders and learning rate variations. The simulation results validate the better performance of the HQFGD over the standard counterpart in terms of estimation accuracy, convergence speed and robustness.

Low light image enhancement based on modified Retinex optimized by fractional order gradient descent with momentum RBF neural network

To dynamically adjust the edge preservation and smoothness of low-light images, this paper proposed a fractional order gradient descent with momentum radial basis function neural network (FOGDMRBF) to optimizing Retinex.Its convergence is proved. In order to speed up the convergence process, an adaptive learning rate is used to adjust the training process reasonably. The results verify the theoretical results of the proposed algorithm such as its monotonicity and convergence. The descending curve of error values by FOGDM is more smoother than gradient descent and gradient descent with momentum method. The influence of regularization parameter is analyzed and compared. Compared with Dark Channel Prior, Histogram Equalization, Homomorphic Filtering and Multiple Exposure Fusion, the halo and noise generated are significantly reduced with higher Peak Signal-to-Noise Ratio and Structural Similarity Index.

Meta Balanced Network for Fair Face Recognition.

Although deep face recognition has achieved impressive progress in recent years, controversy has arisen regarding discrimination based on skin tone, questioning their deployment into real-world scenarios. In this paper, we aim to systematically and scientifically study this bias from both data and algorithm aspects. First, using the dermatologist approved Fitzpatrick Skin Type classification system and Individual Typology Angle, we contribute a benchmark called Identity Shades (IDS) database, which effectively quantifies the degree of the bias with respect to skin tone in existing face recognition algorithms and commercial APIs. Further, we provide two skin-tone aware training datasets, called BUPT-Globalface dataset and BUPT-Balancedface dataset, to remove bias in training data. Finally, to mitigate the algorithmic bias, we propose a novel meta-learning algorithm, called Meta Balanced Network (MBN), which learns adaptive margins in large margin loss such that the model optimized by this loss can perform fairly across people with different skin tones. To determine the margins, our method optimizes a meta skewness loss on a clean and unbiased meta set and utilizes backward-on-backward automatic differentiation to perform a second order gradient descent step on the current margins. Extensive experiments show that MBN successfully mitigates bias and learns more balanced performance for people with different skin tones in face recognition. The proposed datasets are available at http://www.whdeng.cn/RFW/index.html.

A quasi fractional order gradient descent method with adaptive stepsize and its application in system identification

In this paper, the fractional order gradient method (FOGM) is extended to the solution of high-dimensional function optimization problems. A quasi fractional order gradient descent method (QFOGDM) is proposed and then introduce an adaptive stepsize into QFOGDM. The theoretic analysis for convergence of QFOGDM is be done by three theorems. The numerical experiments for solving 15 unconstrained optimization benchmarks are compared to show its’ better performance. Meanwhile, the proposed algorithm is utilized to identify the parameters in the linear discrete deterministic systems and achieves a better convergence rate and accuracy.

Data classification based on fractional order gradient descent with momentum for RBF neural network

ABSTRACT The weight-updating methods have played an important role in improving the performance of neural networks. To ameliorate the oscillating phenomenon in training radial basis function (RBF) neural network, a fractional order gradient descent with momentum method for updating the weights of RBF neural network (FOGDM-RBF) is proposed for data classification. Its convergence is proved. In order to speed up the convergence process, an adaptive learning rate is used to adjust the training process. The Iris data set and MNIST data set are used to test the proposed algorithm. The results verify the theoretical results of the proposed algorithm such as its monotonicity and convergence. Some non-parametric statistical tests such as Friedman test and Quade test are taken for the comparison of the proposed algorithm with other algorithms. The influence of fractional order, learning rate and batch size is analysed and compared. Error analysis shows that the algorithm can effectively accelerate the convergence speed of gradient descent method and improve its performance with high accuracy and validity.

Weighted Internal Model Control-Proportional Integral Derivative Control Scheme Via Fractional Gradient Descent Algorithm

Abstract An adaptive controller design technique based on internal model control (IMC) scheme is proposed in this paper. Multiple IMC controllers having different values of filter time constants and exhibiting widely different performance are combined via weight update rule. The weight update rule, formulated via convex combination of integral and fractional order gradient descent algorithms, assigns time varying weights to individual candidate controllers to obtain an enhanced performance over the individual candidate controllers. The beauty of the proposed technique is that it employs the simplicity of one degree-of-freedom (1DOF) IMC structure to achieve an improved performance over existing 2DOF control schemes. The efficacy of the proposed technique is demonstrated via three illustrative examples and via experimental validation on the hardware setup of dc servosystem. An extensive comparative analysis in terms of simulation plots and performance indices offers a testimony to the effectiveness of the proposed scheme.

Ship Traffic Flow Prediction Based on Fractional Order Gradient Descent with Momentum for RBF Neural Network

To prevent the maritime traffic accidents and make scientific decision, scientific and accurate prediction of the traffic flow is useful, which has often been made by neural network. The weight updating methods have played an important role in improving the performance of neural networks. To ameliorate the oscillating phenomenon in training radial basis function (RBF) neural network, a fractional order gradient descent (GD) with momentum method for updating the weights of RBF neural network (FOGDM-RBF) is proposed. Its convergence is proved. The new algorithm is used to predict vessel traffic flow at Xiamen Port. It performs stable and converges to zero as the iteration increases. The results verify the theoretical results of the proposed algorithm such as its monotonicity and convergence. The descending curve of error values by fractional order GDM is smoother than the GD and GDM method. Error analysis shows that the algorithm can effectively accelerate the convergence speed of the GD method and improve its performance with high accuracy and validity. The influence of fractional order, number of hidden layer neurons, tide peak hours, and ship size is analyzed and compared. 1. Introduction As the world shipping becomes more and more busy, the large ship traffic flow leads to frequent maritime traffic accidents, resulting in huge economic losses. Ship traffic flow is a basic system in marine traffic engineering and an important index to measure the construction of marine traffic infrastructure. Its prediction results can provide basis for formulating scientific Port management planning and ship navigation management. Therefore, to ensure the accuracy and rationality of ship traffic flow forecasting is of great significance for improving port infrastructure construction and formulating scientific port management strategies. Many advanced artificial intelligence optimization algorithms have been used for ship traffic flow forecasting, such as artificial neural network (Zhai 2013; Zhang 2015). Neural network can deal with complex nonlinear problems and has achieved some results. However, the neural network itself has some shortcomings, such as slow learning speed, easy to fall into the local extremum, learning and memory instability, etc.

An innovative fractional order LMS based on variable initial value and gradient order

This article presents a novel fractional order LMS (FOLMS) algorithm, which involves a variable gradient order scheme. The fractional order gradient descent method is revisited firstly. A variable initial value scheme is proposed to attenuate the non–locality of fractional order calculus and to ensure the convergence of the proposed FOLMS algorithm. Furthermore, it is noticed that a contradiction between rapidity and accuracy always appears together with the advancement of FOLMS algorithm; namely, a larger value of the gradient order can not only give a faster convergence speed, but also correspond to a larger estimation error. For the purpose of removing the contradiction between rapidity and accuracy, a variable gradient order scheme is designed for the FOLMS algorithm. Based on a sufficient large number of independent runs, the efficiency and superiority of the proposed algorithm are demonstrated in numerical examples finally.

Medical image registration by combining global and local information: a chain-type diffeomorphic demons algorithm

The demons algorithm is a popular algorithm for non-rigid image registration because of its computational efficiency and simple implementation. The deformation forces of the classic demons algorithm were derived from image gradients by considering the deformation to decrease the intensity dissimilarity between images. However, the methods using the difference of image intensity for medical image registration are easily affected by image artifacts, such as image noise, non-uniform imaging and partial volume effects. The gradient magnitude image is constructed from the local information of an image, so the difference in a gradient magnitude image can be regarded as more reliable and robust for these artifacts. Then, registering medical images by considering the differences in both image intensity and gradient magnitude is a straightforward selection. In this paper, based on a diffeomorphic demons algorithm, we propose a chain-type diffeomorphic demons algorithm by combining the differences in both image intensity and gradient magnitude for medical image registration. Previous work had shown that the classic demons algorithm can be considered as an approximation of a second order gradient descent on the sum of the squared intensity differences. By optimizing the new dissimilarity criteria, we also present a set of new demons forces which were derived from the gradients of the image and gradient magnitude image. We show that, in controlled experiments, this advantage is confirmed, and yields a fast convergence.