Adaptive Gradient Descent Algorithm Research Articles

Physics-informed neural network based on a new adaptive gradient descent algorithm for solving partial differential equations of flow problems

Physics-informed neural network (PINN) is an emerging technique for solving partial differential equations (PDEs) of flow problems. Due to the advantage of low computational cost, the gradient descent algorithms coupled with the weighted objectives method are usually used to optimize loss functions in the PINN training. However, the interaction mechanisms between gradients of loss functions are not fully clarified, leading to poor performances in loss functions optimization. For this, an adaptive gradient descent algorithm (AGDA) is proposed based on the interaction mechanisms analyses and then validated by analytical PDEs and flow problems. First, the interaction mechanisms of loss functions gradients in the PINN training based on the traditional Adam optimizer are analyzed. The main factors responsible for the poor performances of the Adam optimizer are identified. Then, a new AGDA optimizer is developed for the PINN training by two modifications: (1) balancing the magnitude difference of loss functions gradients and (2) eliminating the gradient directions conflict. Finally, three types of PDEs (elliptic, hyperbolic, and parabolic) and four viscous incompressible flow problems are selected to validate the proposed algorithm. It is found that to reach the specified accuracy, the required training time of the AGDA optimizer is about 16%–90% of the Adam optimizer and 41%–64% of the PCGrad optimizer, and the demanded number of iterations is about 10%–68% of the Adam optimizer and 38%–77% of the PCGrad optimizer. Therefore, the PINN method coupled with the AGDA optimizer is a more efficient and robust technique for solving partial differential equations of flow problems.

Solving Electromagnetic Inverse Problem Using Adaptive Gradient Descent Algorithm

The nonlinear conjugate gradient (NLCG) algorithm is one of the popular linearized methods used to solve the frequency-domain electromagnetic (EM) geophysical inverse problem. During NLCG iterations, the model gradient guides the searching direction while the line-search algorithm determines the step length of each iteration. Normally, the line search requires solving the corresponding forward problem a few times. Since line search is usually computationally inefficient, we introduce the adaptive gradient descent (AGD) algorithm to accelerate solving the frequency-domain EM inverse problem within the linearized framework. The AGD algorithm is a variant of the classical gradient descent method and has been well-developed and widely used in deep learning. Rather than the time-consuming line search, its core idea is to algebraically manipulate the cumulative gradients and updates of the model from previous iterations to estimate the model parameter variables at the current iteration. For the inversion of magnetotelluric (MT) data, we here designed and implemented a framework using the AGD algorithm combined with the cool-down scheme to tune the regularization parameter. To improve the convergence performance of the AGD algorithm [specifying to Adam and root-mean-square propagation (RMSProp)], we proposed a tolerance strategy which has been tested numerically. To optimize the global learning rate, we carried out some comparative trials in the proposed inversion framework. The inverted results of synthetic and real-world data showed that both the AGD algorithms (Adam and RMSProp) can recover comparable results and save more than a third of CPU time compared with the NLCG algorithm.

An adaptive gradient descent attitude estimation algorithm based on a fuzzy system for UUVs

An adaptive gradient descent algorithm (AGDA) based on a fuzzy system is proposed to improve the attitude estimation accuracy and adaptability of unmanned underwater vehicles (UUVs) under various ocean environments. First, the algorithm uses current and historical gyroscope data to predict the quaternion of the current moment. Then, the accelerometer data are calibrated according to the predicted quaternion, and the gradient direction is obtained. At the same time, a fuzzy system generated by the genetic algorithm is used to solve the adaptive coefficient. Finally, the algorithm uses the gradient descent method to achieve data fusion, compensating for the accumulated error of the gyroscope. The experimental results show that the prediction of the quaternion and the adaptive coefficient can effectively improve the accuracy and environmental adaptability of the algorithm, and the proposed algorithm has better dynamic and static performances than traditional attitude estimation algorithms.

Medical image fusion via discrete stationary wavelet transform and an enhanced radial basis function neural network

Medical image fusion of images obtained via different modes can expand the inherent information of original images, whereby the fused image has a superior ability to display details than the original sub-images, to facilitate diagnosis and treatment selection. In medical image fusion, an inherent challenge is to effectively combine the most useful information and image details without information loss. Despite the many methods that have been proposed, the effective retention and presentation of information proves challenging. Therefore, we proposed and evaluated a novel image fusion method based on the discrete stationary wavelet transform (DSWT) and radial basis function neural network (RBFNN). First, we analyze the details or feature information of two images to be processed by DSWT by using two-level decomposition to separate each image into seven parts, comprising both high-frequency and low-frequency sub-bands. Considering the gradient and energy attributes of the target, we substituted the pending parts in the same position in the two images by using the proposed enhanced RBFNN. The input, hidden, and output layers of the neural network comprised 8, 40, and 1 neuron(s), respectively. From the seven neural networks, we obtained seven fused parts. Finally, through inverse wavelet transform, we obtained the final fused image. For the neural network training method, the hybrid adaptive gradient descent algorithm (AGDA) and gravitational search algorithm (GSA) were implemented. The final experimental results revealed that the novel method has significantly better performance than the current state-of-the-art methods.

A self-adaptive gradient descent search algorithm for fully-connected neural networks

Artificial neural networks have achieved great success in artificial intelligence. Back propagation is one of the most widely used parameter search techniques for artificial neural networks, and it has been widely used in many deep learning approaches such as convolutional neural networks (CNNs), generator adversarial networks (GANs) and recurrent neural networks (RNNs) to train their fully-connected layers. However, the fully-connected layers of the neural networks with back propagation technique depends on the gradient descent strategy, which has the disadvantage of easily falling into local optima. Meanwhile, many evolutionary computation (EC) techniques are applied to the parameters optimization of neural networks. They take advantage of populations to achieve global search. But their convergence speed is relatively slow since they use their original evolutionary search strategies. In this paper, we propose a self-adaptive gradient descent search algorithm (SaGDSA) to search the parameters for fully-connected neural networks. Different from the existing adaptive gradient descent algorithm, the proposed algorithm does not require the user to have any prior knowledge and manually design the learning rate of different stages to match the different search stages. In addition, four kinds of gradient descent strategies are used to optimize parameters of fully-connected neural networks. All used datasets are collected from the University of California Irvine (UCI) machine learning repository. The experimental results indicate that SaGDSA is better than the comparison algorithms on most of the datasets. By introducing the self-adaptive mechanism, SaGDSA not only has excellent global search capabilities from evolutionary computation but also has local search capabilities from gradient descent.

Biased compensation adaptive gradient algorithm for rational model with time-delay using self-organising maps

This paper develops a biased compensation adaptive gradient decent algorithm for rational models with unknown time-delay. Owing to the unknown time-delay, traditional identification methods cannot be directly applied for such models. To overcome this difficulty, the self-organised maps are proposed, which can obtain the estimates of the time-delay based on the residual errors. Then, an adaptive gradient descent algorithm is introduced to obtain the parameter estimates. Compared with the traditional gradient descent and redundant rule based methods, the proposed method has two advantages: (1) each element in the parameter vector has its own step-size, thus it is more effective than the traditional gradient descent method; (2) the number of the unknown parameters is unchanged, therefore, it has less computational effort than the redundant rule based method. Finally, a simulation experiment is given to show the effectiveness of the proposed algorithm.

Coordinated control system modeling of ultra-supercritical unit based on a new fuzzy neural network

The coordinated control systems (CCS) in ultra-supercritical thermal power unit, like many other industrial systems, is a complex multivariable system with severe nonlinearity, strong multivariable coupling and uncertainties. In order to meet the requirements of operational stability, economy. etc in ultra-supercritical unit, it is necessary to establish its accurate mathematical model and further design the advanced controller. Against this background, a new fuzzy neural network modeling method is proposed in this paper. First of all, the incremental model is considered separately to improve the rationality of the local linear model structure. Then, the parameters in antecedent part is initialized by a kernel k-means++ algorithm, in which Xie-Beni index is used to optimize the number of fuzzy rules. Finally, supervised adaptive gradient descent algorithm and artificial immune particle swarm optimization algorithm work in stages to complete the training of the consequent part parameters. The proposed modeling method in this paper is applied to a 1000 MW unit in China and shows satisfactory accuracy. In the established model, the MSE of power output, main steam pressure and separator outlet steam temperature are 0.0099, 1.21E-4, 0.0023, respectively. Both numerical and graphical simulation results confirm the effectiveness of the presented fuzzy neural network in modeling.

Adaptive Gradient Descent Algorithm for Networked Control Systems Using Redundant Rule

This study proposes an adaptive gradient descent algorithm for networked control systems, where the systems encounter a time-delay. To estimate the parameters and time-delay simultaneously, a redundant rule based method is introduced to transform the system into an augmented model. Then, the parameters are estimated by using an adaptive gradient descent algorithm. Finally, the true parameters and time-delay can be obtained based on the parameter estimates of the augmented model. A simulation example is proposed to validate the effectiveness of the proposed algorithm.

Improved Pedestrian Positioning with Inertial Sensor Based on Adaptive Gradient Descent and Double-Constrained Extended Kalman Filter

The Foot-mounted Inertial Pedestrian-Positioning System (FIPPS) based on the Micro-Inertial Measurement Unit (MIMU) is a good choice for the forest fire fighters when the Global Navigation Satellite System is unavailable. Zero Velocity Update (ZUPT) provides a solution for reducing cumulative positioning errors caused by the integral calculation of the inertial navigation. However, the performance of ZUPT is highly affected by the low accuracy and high noise of the MIMU. The accuracy of conventional ZUPT for attitude alignment is reduced by the zero offset of acceleration and the drift of a gyroscope during the standing phase. An initial alignment algorithm based on Adaptive Gradient Descent Algorithm (AGDA) is proposed. In the stepping phase, the extended Kalman filter (EKF) is often used to correct attitude and position in track estimation. However, the measurement noise of the EKF is influenced by the high-frequency acceleration and angular velocity. Thus, the accuracy of the attitude and position will decrease. A double-constrained extended Kalman filtering (DEKF) is proposed. An adaptive parameter positively correlated with the acceleration and angular velocity is set, and the measurement noise in the DEKF is adaptively adjusted. The performance of the proposed method is verified by implementing the pedestrian test trajectory using MPU-9150 MIMU manufactured by InvenSense. The results show that the attitude error of the AGDA is 33.82% less than that of the conventional GDA. The attitude error of DEKF is 21.70% less than that of the conventional EKF. The experimental results verify the effectiveness and applicability of the proposed method.

Removal of computed tomography ring artifacts via radial basis function artificial neural networks

Ring artifacts in computed tomography (CT) images are caused by the undesirable response of detector pixels, which leads to the degradation of CT images. Accordingly, it affects the image interpretation, post-processing, and quantitative analysis. In this study, a radial basis function neural network (RBFNN) was used to remove ring artifacts. The proposed method employs polar coordinate transformation. First, ring artifacts were transformed into linear artifacts by polar coordinate transformation. Then, smoothing operators were applied to locate these artifacts exactly. Subsequently, RBFNN was operated on each linear artifact. The neuron numbers of the input, hidden, and output layers of the neural network were 8, 40, and 1, respectively. Neurons in the input layer were selected according to the characteristics of the artifact itself and its relationship with the surrounding normal pixels. For the training of the neural network, a hybrid of adaptive gradient descent algorithm (AGDA) and gravitational search algorithm (GSA) was adopted. After the corrected image was obtained using the updated neural network, the inverse coordinate transformation was implemented. The experimental data were divided into simulated ring artifacts and real ring artifacts, which were based on brain and abdomen CT images. Compared with current artifact removal methods, the proposed method removed ring artifacts more effectively and retained the maximum detail of normal tissues. In addition, for index analysis, the performance of proposed method was superior to that of the other methods.

Research on an Improved Method for Foot-Mounted Inertial/Magnetometer Pedestrian-Positioning Based on the Adaptive Gradient Descent Algorithm.

Foot-mounted Inertial Pedestrian-Positioning Systems (FIPPSs) based on Micro Inertial Measurement Units (MIMUs), have recently attracted widespread attention with the rapid development of MIMUs. The can be used in challenging environments such as firefighting and the military, even without augmenting with Global Navigation Satellite System (GNSS). Zero Velocity Update (ZUPT) provides a solution for the accumulated positioning errors produced by the low precision and high noise of the MIMU, however, there are some problems using ZUPT for FIPPS, include fast-initial alignment and unobserved heading misalignment angle, which are addressed in this paper. Our first contribution is proposing a fast-initial alignment algorithm for foot-mounted inertial/magnetometer pedestrian positioning based on the Adaptive Gradient Descent Algorithm (AGDA). Considering the characteristics of gravity and Earth’s magnetic field, measured by accelerometers and magnetometers, respectively, when the pedestrian is standing at one place, the AGDA is introduced as the fast-initial alignment. The AGDA is able to estimate the initial attitude and enhance the ability of magnetic disturbance suppression. Our second contribution in this paper is proposing an inertial/magnetometer positioning algorithm based on an adaptive Kalman filter to solve the problem of the unobserved heading misalignment angle. The algorithm utilizes heading misalignment angle as an observation for the Kalman filter and can improve the accuracy of pedestrian position by compensating for magnetic disturbances. In addition, introducing an adaptive parameter in the Kalman filter is able to compensate the varying magnetic disturbance for each ZUPT instant during the walking phase of the pedestrian. The performance of the proposed method is examined by conducting pedestrian test trajectory using MTi-G710 manufacture by XSENS. The experimental results verify the effectiveness and applicability of the proposed method.

Hybrid Analog and Digital Precoding Based on Adaptive Method in P2P Massive MIMO

To achieve a far higher capacity in massive multi-input multi-output, the hybrid analog and digital precoding has been studied recently. In this paper, we consider the downlink single user communication system and propose an efficient matrix adaptive gradient descent algorithm to design the radio frequency precoder and combiner, and use the singular value decomposition method to obtain the digital precoder and combiner. Simulation results show that the proposed method can approach the performance of the conventional full digital precoding with a fast convergence rate.

Designing stable neural identifier based on Lyapunov method

The stability of learning rate in neural network identifiers and controllers is one of the challenging issues, which attract many researchers’ interest in neural networks. This paper suggests adaptive gradient descent algorithm with stable learning laws for modified dynamic neural network (MDNN) and studies the stability of this algorithm. Also, stable learning algorithm for parameters of MDNN is proposed. By the proposed method, some constraints are obtained for learning rate. Lyapunov stability theory is applied to study the stability of the proposed algorithm. The Lyapunov stability theory guaranteed the stability of the learning algorithm. In the proposed method, the learning rate can be calculated online and will provide an adaptive learning rate for the MDNN structure. Simulation results are given to validate the results.