Traditional Gradient Descent Method Research Articles

Novel Step Size for Unconstrained Optimization

The step length size is the basis of gradient descent method with the desirable conjugate property. The aim of this document is to present a fresh gradient descent technique that calculates the step size based on a basic estimation of the Hessian of the function being minimized. The algorithm that results falls within the category of gradient descent methods with linear convergence characteristics. Our study includes numerical data that demonstrates the significant advantages of our approach compared to traditional gradient descent methods.

Prediction of Health Status of Small-Tailed Cold Sheep Based on Improved BP Neural Network

According to related research, different body temperatures, heart rates, and locomotor behaviors of small-tailed cold goats can represent the physical condition of the goats themselves and are used as direct evidence for evaluating the physical health status of small-tailed cold goats. In this paper, we designed and tested a system for predicting the health status of small-tailed cold sheep based on wearable information monitoring technology. To test the system, sheep wearable devices were worn on 36 small-tailed cold sheep of different ages and inconsistent health conditions at different time points from May to October. A SLBAS-BP neural network model for predicting the health condition of small-tailed cold sheep was established using the collected and processed data, which overcame the problem that the traditional gradient descent method in the BP neural network is prone to fall into local optimization leading to insufficient prediction ability. The correct prediction rates of the improved BP neural network for the four health conditions of healthy, sub-healthy, fever, and disease were 98.4%, 94.5%, 90.4%, and 98.7%, respectively, and the average correct prediction rate of the four conditions was 5.8% higher than that before the improvement, reaching 95.2%.

Optimizing Parameter Efficiency in Machine Learning Models: A Focus on Reducing Memory Overhead with L-BFGS-Optimized Algorithms

This research addresses the critical challenge of detecting cotton leaf diseases through an advanced image processing and machine learning approach. The study primarily focuses on the extraction of significant features from cotton leaf images to facilitate accurate disease identification. The novelty of this work lies in the application of the Limited-memory Broyden–Fletcher–Goldfarb–Shanno (L-BFGS) algorithm for optimizing Support Vector Machines (SVM), enhancing the precision of disease detection in cotton leaves. This research method involves a comprehensive process of acquiring high-resolution leaf images, followed by an effective feature extraction technique that isolates crucial characteristics indicative of various plant diseases. Subsequently, these features are employed to train an SVM model optimized by the L-BFGS algorithm, a decision process that marks a significant departure from traditional gradient descent methods in SVM training. A pivotal aspect of this study is the comparative analysis conducted between the proposed L-BFGS-optimized SVM model and prevalent machine learning models including Random Forest, Decision Tree, Regression Models, and K-Nearest Neighbors (KNN). This comparison is grounded on various performance metrics such as accuracy, precision, recall, and F1-score, offering a comprehensive evaluation of each model's effectiveness in disease detection. The results of this research are expected to demonstrate the superiority of the L-BFGS-optimized SVM in terms of accuracy and efficiency in detecting cotton leaf diseases. This advancement holds substantial promise for agricultural technology, potentially leading to more informed and timely decision-making in crop management and disease prevention. The findings of this study aim to contribute significantly to the field of agricultural informatics, particularly in the domain of plant disease detection and management, by providing a more robust, accurate, and efficient tool for farmers and agriculturalists.

Relevance gradient descent for parameter optimization of image enhancement

Machine learning plays a pivotal role in constructing learning models and leveraging abundant image data for feature extraction, which is essential for achieving accurate and efficient image enhancement through optimization algorithms. Traditional gradient descent methods are confined to optimizing independent parameters, while recent algorithms neglect the interdependence among parameters, resulting in prolonged iteration times for each method. To address this issue, we propose a novel Correlation-Based Gradient Descent (RGD) algorithm tailored for enhancing low-light images. RGD leverages inter-iteration results for mutual parameter transmission and optimization, employing convergence or oscillation criteria to select the final enhanced image and find the optimal parameter combination for image enhancement, thereby enhancing parameter optimization efficiency. Specifically, each iteration is divided into multiple operations, with each operation adopting different enhancement schemes. During each iteration, the results of different operation parameters are shared, which facilitates the discovery of optimal parameters for low-light image enhancement. Subsequently, corresponding enhancement operations are performed based on this parameter combination to optimize standard indicators. Our experiments on medical and natural low-light scenarios demonstrate that our RGD method achieves a significant reduction in iteration time, with at least a 17% decrease compared to exhaustive loop methods.

Improved SSA‐RBF neural network‐based dynamic 3‐D trajectory tracking model predictive control of autonomous underwater vehicles with external disturbances

AbstractThis paper studies the three‐dimensional (3‐D) dynamic trajectory tracking control of an autonomous underwater vehicle (AUV). As AUV is a typical nonlinear system, each degree of freedom is strongly coupled, so the traditional control method based on the nominal model of AUV cannot guarantee the accuracy of the control system. To solve this problem, we first propose a prediction model based on a radial basis function neural network (RBF‐NN). The nonlinearity of AUV is learned and modeled offline by RBF‐NN based on previous data. This model can reflect the time sequence state and control variables of AUV. Secondly, to avoid the overfitting problem in network training based on the traditional gradient descent method, a new adaptive chaotic sparrow search algorithm (ACSSA) is proposed to optimize the network parameters, to improve the full approximation ability of RBF‐NN to nonlinear systems. To eliminate the steady‐state error caused by external interference during AUV trajectory tracking, a nonlinear optimizer is designed by updating the deviation of the NN model output layer. In each sampling period, the predictive control law is calculated online according to the deviation between the predicted value and the actual value. In addition, the stability analysis based on the Lyapunov method proves the asymptotic stability of the controller. Finally, the 3‐D dynamic trajectory tracking the performance of AUV under different external disturbances is verified by MATLAB/Simulink, and the results show that the proposed controller is more efficient and robust than the standard model predictive controller (MPC) controller and the standard NN model predictive controller (NNPC).

The advance of underdetermined linear regression optimization based on implicit bias

The gradient descent method has the characteristics of easy realization and simple structure. The traditional gradient descent method has many advantages, especially in solving convex optimization problems. In recent years, some researchers have noticed that the gradient descent algorithm is helpful to solve the problem of underdetermined linear regression optimization. Therefore, in order to explore the specific relationship between gradient descent and under-determined linear regression optimization, this paper focuses on a case with a unique finite root loss function and discusses in detail the working principles of the natural-gradient-descent, mirror-descent, and gradient-descent in this case, and proves the auxiliary effect of gradient descent on under-determined linear regression optimization theoretically. How to use the characteristics of the gradient descent algorithm to design a suitable model, and how do we use the idea of the gradient descent algorithm to find an answer for problem of under determined linear regression optimization in the practical application will be the future research work to continue to advance.

Mirror Descent of Hopfield Model

Abstract Mirror descent is an elegant optimization technique that leverages a dual space of parametric models to perform gradient descent. While originally developed for convex optimization, it has increasingly been applied in the field of machine learning. In this study, we propose a novel approach for using mirror descent to initialize the parameters of neural networks. Specifically, we demonstrate that by using the Hopfield model as a prototype for neural networks, mirror descent can effectively train the model with significantly improved performance compared to traditional gradient descent methods that rely on random parameter initialization. Our findings highlight the potential of mirror descent as a promising initialization technique for enhancing the optimization of machine learning models.

Heavy-ball-based hard thresholding algorithms for sparse signal recovery

The hard thresholding technique plays a vital role in the development of algorithms for sparse signal recovery. By merging this technique and heavy-ball acceleration method which is a multi-step extension of the traditional gradient descent method, we propose the so-called heavy-ball-based hard thresholding (HBHT) and heavy-ball-based hard thresholding pursuit (HBHTP) algorithms for signal recovery. It turns out that the HBHT and HBHTP can successfully recover a k-sparse signal if the restricted isometry constant of the measurement matrix satisfies δ3k<0.618 and δ3k<0.577, respectively. The guaranteed success of HBHT and HBHTP is also shown under the conditions δ2k<0.356 and δ2k<0.377, respectively. Moreover, the finite convergence of HBHTP and stability of the two algorithms are also established in this paper. Simulations on random problem instances are performed to compare the performance of the proposed algorithms and several existing ones. Empirical results indicate that the HBHTP performs very comparably to a few existing algorithms and it takes less average time to achieve the signal recovery than these existing methods.

Improved Gradient Descent Iterations for Solving Systems of Nonlinear Equations

This research proposes and investigates some improvements in gradient descent iterations that can be applied for solving system of nonlinear equations (SNE). In the available literature, such methods are termed improved gradient descent methods. We use verified advantages of various accelerated double direction and double step size gradient methods in solving single scalar equations. Our strategy is to control the speed of the convergence of gradient methods through the step size value defined using more parameters. As a result, efficient minimization schemes for solving SNE are introduced. Linear global convergence of the proposed iterative method is confirmed by theoretical analysis under standard assumptions. Numerical experiments confirm the significant computational efficiency of proposed methods compared to traditional gradient descent methods for solving SNE.

Research on site selection of agricultural internet of things nodes based on rapid terrain sampling

With the development of Internet of Things (IoT) technology, modern agriculture is moving in the direction of 4.0. The Agricultural IoT is inseparable from wireless communication. However, in traditional agricultural IoT router and gateway site selection, the influence of the actual terrain on transmission loss is not considered, which results in node power wastage and increased maintenance costs. Based on a multi-sensor fusion algorithm, a fast terrain sampler is designed in this study to collect point-cloud data of the experimental site terrain. A reasonable objective function is then designed under the premise of consideration of the electromagnetic wave free-space and diffraction losses, and the locations of the routers and gateway are optimized based on k-means and particle swarm optimization (PSO) algorithm. Simulations show that the running time of the PSO algorithm is very sensitive to the changes in the execution parameters, and the improved PSO algorithm converges faster than the genetic algorithm (GA) in all three initialization methods. After collecting field terrain data, five interpolation methods were compared, and the nearest-neighbor algorithm is used to obtain the terrain model. On-site collection of received signal-strength indication (RSSI) shows that the communication quality of the optimal point selected via this algorithm is significantly higher than those of nearby points. At the same time, it is proved that the RSSI data has serious discontinuities, so the traditional gradient descent method is not suitable for solving the objective function in this work. Therefore, the algorithm used herein is of great significance for the site selection of agricultural Internet of Things nodes. However, since there are many ways to calculate the diffraction loss, the objective function of this work still needs more correction studies. The tool proposed in this work can obtain a 3D model of farmland faster than traditional surveying and mapping methods. With future research, the 3D model may be applied to soil moisture analysis, rainfall and moisture direction inversion, and plant light exposure prediction.

Research on Medical Information Extraction and Recognition of Lung Tumor Based on Convolutional Neural Network

With the improvement of the social economy, lung cancer has gradually become one of the main reasons for endangering human security. The tumor is benign or malignant, which can reduce the cumbersome examination in the early stage, prevent unnecessary surgery, reduce the psychological and physical pain of patients, and improve the survival rate of patients. As a common method of treatment and detection of treatment response in the medical field, the medical image has become the main technical means of cancer diagnosis and treatment in the clinic. This paper studies the extraction and recognition technology of lung tumor medical information based on a convolutional neural network (CNN), and the method used is CNN. CNN provides an end-to-end learning model. The parameters in the model can be trained by the traditional gradient descent method. The trained CNN can learn the features in the image, and complete the extraction and classification of image features. After research, the algorithm in this paper is effective and suitable for wide use.

Time-Optimal Trajectory Planning of 6-DOF Manipulator Based on Fuzzy Control

Currently, the teaching programming or offline programming used by an industrial manipulator can manually set the running speed of the manipulator. In this paper, to consider the running speed and stability of the manipulator, the time-optimal trajectory planning (TOTP) of the manipulator is transformed into a nonlinear optimal value search problem under multiple constraints, and a time-search algorithm based on fuzzy control is proposed, so that the end of the manipulator can run along the given path in Cartesian space for the shortest time, and the angular velocity and angular acceleration of each joint is within a limited range. In addition, a simulation model of a 6-DOF manipulator is established in MATLAB, taking a straight-line trajectory of the end of the manipulator in Cartesian space as an example, and the effectiveness and efficiency of the algorithm proposed in this paper are proved by comparing the execution time with the bisection algorithm and the traditional gradient descent method.

Optimal trajectory-tracking guidance for reusable launch vehicle based on adaptive dynamic programming

An optimal trajectory-tracking guidance method for Reusable Launch Vehicle (RLV) based on neural Adaptive Dynamic Programming (ADP) is proposed. Firstly, a reentry reference trajectory and the corresponding steady-state control are generated based on the Gauss Pseudo-spectral Method (GPM) with saturation constraints on the amplitude, rate and acceleration of flow angles. A single-critic ADP controller is designed for optimal feedback control, which is combined with the steady-state control to realize the trajectory-tracking guidance. An innovative weight iteration algorithm for the critic neural network is proposed to reduce training computation and improve guidance accuracy. Simulation results show that with initial state errors and system uncertainty, the terminal position error is within 2.1 km in a reentry flight whose range covers more than 5500 km. Involved with the innovative weight iteration method, the training computation is reduced by 95% compared with the traditional gradient descent method and the guidance performance is also improved.

Biased compensation adaptive gradient algorithm for rational model with time-delay using self-organising maps

This paper develops a biased compensation adaptive gradient decent algorithm for rational models with unknown time-delay. Owing to the unknown time-delay, traditional identification methods cannot be directly applied for such models. To overcome this difficulty, the self-organised maps are proposed, which can obtain the estimates of the time-delay based on the residual errors. Then, an adaptive gradient descent algorithm is introduced to obtain the parameter estimates. Compared with the traditional gradient descent and redundant rule based methods, the proposed method has two advantages: (1) each element in the parameter vector has its own step-size, thus it is more effective than the traditional gradient descent method; (2) the number of the unknown parameters is unchanged, therefore, it has less computational effort than the redundant rule based method. Finally, a simulation experiment is given to show the effectiveness of the proposed algorithm.

Intelligent injection molding: Parameters self‐learning optimization using iterative gradient‐approximation adaptive method

AbstractIntelligent injection molding consists of three aspects: intelligent parameter optimization, process monitoring and control. The optimal process parameters are critical to guarantee product quality. Injection molding is a typical batch process and has the property that previous runs can provide feedback to optimize subsequent runs. This study proposes a self‐learning parameter optimization method named iterative gradient‐approximation adaptive optimization (IGAO) method, which adopts the batch‐to‐batch information to remove the need to establish optimization model with large numbers of experiments. The analysis and the optimization of the parameters can be performed simultaneously. The IGAO method approximates the gradient iteratively and assigns an adaptive step size to each parameter according to gradient accumulation. The experiments conducted in both simulation software and injection molding machine prove that the method has fast convergence speed. Standard product weight can be obtained within 11 runs from three different starting process parameters. Experiment results show that 25% less steps are needed compared with the traditional gradient descent method. The method also has good stability to resist disturbances during the optimization procedure. In general, the proposed IGAO method is fast, stable and robust, and it has good prospects for parameter optimization in batch processes.

Unsupervised machine fault diagnosis for noisy domain adaptation using marginal denoising autoencoder based on acoustic signals

Recently, with the desperate demand for data-driven deep learning methods in practical industrial applications, increasing popularity of deep learning methods for machine fault diagnosis under noisy environments has been observed. However, the existing studies have encountered two limitations: 1) the domain shift problem has not been researched enough to ensure a functional adaptation model across different noisy domains; 2) those deep learning frameworks are generally complicated and require massive computational space. To address those problems, we propose a Noisy Domain Adaptive marginal Stacking Denoising Auto-encoder (NDAmSDA) based on acoustic signals for domain adaptation between different noise levels. Specifically, a modified mSDA, integrated with a dimensionality reduction method based on Transfer Component Analysis (TCA), not only facilitates the training acceleration by replacing the traditional gradient descent method of backpropagation with a forward closed-form solution, but also promotes gapping the discrepancy between various noise levels and transferring the classifier built from one noisy domain to others. This enables us to extend diagnostic models’ capability to real-world scenarios because of the unavoidable background noise in the collected signals. To validate the effectiveness of our proposed approach, datasets mixed with additive white Gaussian noise (AWGN) and binary masking noise (BMN) from the gear tests and motor tests are constructed for validation. Comparisons with representative and popular fault diagnosis approaches confirm the superiority of the proposed method.

Prediction of the Slope Solute Loss Based on BP Neural Network

The existence of soil macropores is a common phenomenon. Due to the existence of soil macropores, the amount of solute loss carried by water is deeply modified, which affects watershed hydrologic response. In this study, a new improved BP (Back Propagation) neural network method, using Levenberg–Marquand training algorithm, was used to analyze the solute loss on slopes taking into account the soil macropores. The rainfall intensity, duration, the slope, the characteristic scale of macropores and the adsorption coefficient of ions, are used as the variables of network input layer. The network middle layer is used as hidden layer, the number of hidden nodes is five, and a tangent transfer function is used as its neurons transfer function. The cumulative solute loss on the slope is used as the variable of network output layer. A linear transfer function is used as its neurons transfer function. Artificial rainfall simulation experiments are conducted in indoor experimental tanks in order to verify this model. The error analysis and the performance comparison between the proposed method and traditional gradient descent method are done. The results show that the convergence rate and the prediction accuracy of the proposed method are obviously higher than that of traditional gradient descent method. In addition, using the experimental data, the influence of soil macropores on slope solute loss has been further confirmed before the simulation.

Application of supervised descent method for 2D magnetotelluric data inversion

The supervised descent method (SDM) is applied to 2D magnetotellurics (MT) data inversion. SDM contains offline training and online prediction. The training set is composed of the models generated according to prior knowledge and the data simulated by MT forward modeling. In the training process, a set of descent directions from an initial model to the training models is learned. In the prediction, model reconstruction is achieved by optimizing an online regularized objective function with a restart scheme, where the learned descent directions and the computed data residual are involved. SDM inversion has the advantages of (1) being more efficient than traditional gradient-descent methods because the computation of local derivatives of the objective function is avoided, (2) incorporating prior uncertain knowledge easier than deterministic inversion approach by generating training models flexibly, and (3) having high generalization ability because the physical modeling can guide the online model reconstruction. Furthermore, a way of designing general training set is introduced, which can be used for training when the prior knowledge is weak. The efficiency and accuracy of this method are validated by two numerical examples. The results indicate that the reconstructed models are consistent with prior information, and the simulated responses agree well with the data. This method also shows good potential to improve the accuracy and efficiency in field MT data inversion.

A fast computational approach for illusory contour reconstruction

Illusory contour reconstruction can be modeled as a minimization problem with a tractable variational level set formulation, utilizing Euler’s elastica to reconstruct the illusory boundaries. However, this kind of formulation is very difficult to solve numerically as it is hard to implement such optimization algorithms efficiently in practice. In this paper, we propose an equivalently reduced variational level set formulation by taking the level set functions as signed distance functions. Technically, an alternating direction method of multipliers (ADMM) is developed by introducing some auxiliary variables, Lagrange multipliers and applying an alternating optimization strategy. With the proposed ADMM method, the minimization problem can be transformed into a series of sub-problems, which can be solved easily via using the Gauss-Seidel iterations and Fast Fourier Transform (FFT). The corresponding level set functions are regarded as signed distance functions during computation process using a simple algebraic projection method, which avoids the traditional re-initialization process for conventional level set functions. Extensive experiments have been conducted on both synthetic and real images, which validated the proposed approach, and demonstrated the advantages of the proposed ADMM-Projection (ADMM-P) method over the existing algorithms based on traditional gradient descent method (GDM) in terms of computational efficiency.

Color image restoration and inpainting via multi-channel total curvature

The multi-channel total variation (MTV) based on L2 norm is capable of preserving object edges and smoothing flat regions in color images. However, it will lead to loss of image contrast, smear object corners, and produce staircase artifacts in the restored images. In order to remedy these side effects, we propose a new multi-channel total curvature model based on L1 norm (MTC-L1) for vector-valued image restoration in this paper. By introducing some auxiliary variables and Lagrange multipliers, we develop a fast algorithm based alternating direction method of multipliers (ADMM) for the proposed model, which allows the use of the fast Fourier transform (FFT), generalized soft threshold formulas and projection method. Extensive experiments have been conducted on both synthetic and real color images, which validate the proposed approach for better restoration performance, and show advantages of the proposed ADMM over algorithms based on traditional gradient descent method (GDM) in terms of computational efficiency.