Davidon-Fletcher-Powell Research Articles

Enhancing Support Vector Machine Performance: A Hybrid Approach with Davidon-Fletcher-Powell Algorithm and Elephant Herding Optimization (EHO-DFP) for Parameter Optimization

Support Vector Machines (SVMs) have gained prominence in machine learning for their capability to establish optimal decision boundaries in high-dimensional spaces. SVMs are powerful machine learning models but can encounter difficulties in achieving optimal performance due to challenges such as selecting appropriate kernel parameters, handling uncertain data, and adapting to complex decision boundaries.. This paper introduces a novel hybrid approach to enhance the performance of Support Vector Machines (SVM) through the integration of the Davidon-Fletcher-Powell (DFP) optimization algorithm and Elephant Herding Optimization (EHO) for parameter tuning. SVM, a robust machine learning algorithm, relies on effective hyperparameter selection for optimal performance. The proposed hybrid model synergistically leverages DFP's efficiency in unconstrained optimization and EHO's exploration-exploitation balance inspired by elephant herding behavior. The fusion of these algorithms address the challenges associated with traditional optimization methods. The hybrid model offers improved convergence towards the global optimum. Experimental results demonstrate the efficacy of the approach, showcasing enhanced SVM performance in terms of minimum 3.3% accuracy and 3.4% efficiency. This research contributes to advancing the field of metaheuristic optimization in machine learning, providing a promising avenue for effective parameter optimization in SVM applications.

A modified quasi-Newton method for uncertain multiobjective optimization problems under a finite uncertainty set

ABSTRACT In this article, a modified quasi-Newton method is developed for the robust counterpart (RC) of an uncertain multiobjective optimization problem (UMOP). Specifically, the RC of a UMOP is the minimum of an objective-wise worst-case (OWWC) type RC. To determine the descent direction for the RC, a subproblem using a common Hessian approximation is constructed. An inexact line search technique based on the Armijo technique is employed to find an appropriate step length. Furthermore, a modified version of the Davidon–Fletcher–Powell (DFP) update formula is developed for the RC. A modified quasi-Newton algorithm is built using descent direction step length, and DFP Hessian update formula. Furthermore, the convergence of the modified quasi-Newton algorithm is also discussed. Finally, the modified quasi-Newton algorithm is illustrated with a set of test problems and compared with the weighted sum method. Moreover, it is demonstrated that the algorithm is effective for both convex and non-convex problems.

Machine Learning in Quasi-Newton Methods

In this article, we consider the correction of metric matrices in quasi-Newton methods (QNM) from the perspective of machine learning theory. Based on training information for estimating the matrix of the second derivatives of a function, we formulate a quality functional and minimize it by using gradient machine learning algorithms. We demonstrate that this approach leads us to the well-known ways of updating metric matrices used in QNM. The learning algorithm for finding metric matrices performs minimization along a system of directions, the orthogonality of which determines the convergence rate of the learning process. The degree of learning vectors’ orthogonality can be increased both by choosing a QNM and by using additional orthogonalization methods. It has been shown theoretically that the orthogonality degree of learning vectors in the Broyden–Fletcher–Goldfarb–Shanno (BFGS) method is higher than in the Davidon–Fletcher–Powell (DFP) method, which determines the advantage of the BFGS method. In our paper, we discuss some orthogonalization techniques. One of them is to include iterations with orthogonalization or an exact one-dimensional descent. As a result, it is theoretically possible to detect the cumulative effect of reducing the optimization space on quadratic functions. Another way to increase the orthogonality degree of learning vectors at the initial stages of the QNM is a special choice of initial metric matrices. Our computational experiments on problems with a high degree of conditionality have confirmed the stated theoretical assumptions.

Wireless sensor node localization algorithm combined with PSO-DFP

Abstract In wireless communication technology, wireless sensor networks usually need to collect and process information in very harsh environment. Therefore, accurate positioning of sensors becomes the key to wireless communication technology. In this study, Davidon–Fletcher–Powell (DFP) algorithm was combined with particle swarm optimization (PSO) to reduce the influence of distance estimation error on positioning accuracy by using the characteristics of PSO iterative optimization. From the experimental results, among the average precision (AP) values of DFP, PSO, and PSO-DFP algorithms, the AP value of PSO-DFP was 0.9972. In the analysis of node positioning error, the maximum node positioning error of PSO-DFP was only about 21 mm. The results showed that the PSO-DFP algorithm had better performance, and the average positioning error of the algorithm was inversely proportional to the proportion of anchor nodes, node communication radius, and node density. In conclusion, the wireless sensor node location algorithm combined with PSO-DFP has a better location effect and higher stability than the traditional location algorithm.

A Note on Weibull Parameter Estimation with Interval Censoring Using the EM Algorithm

In many engineering applications, it is often the case that the observations are only available in interval form. In this note, by using the expectation-maximization (EM) algorithm, the parameter estimation of the Weibull distribution with interval-censored data is considered. The estimates obtained using the EM algorithm are compared with those obtained using the conventional Newton-type methods, including the Davidon–Fletcher–Powell (DFP) and Berndt–Hall–Hall–Hausman (BHHH) methods. The results indicate that the estimates obtained using the proposed EM method demonstrate superior convergence properties compared to the conventional DFP and BHHH methods. Finally, a numerical study that illustrates the advantages of the proposed method is provided.

A One-Parameter Memoryless DFP Algorithm for Solving System of Monotone Nonlinear Equations with Application in Image Processing

In matrix analysis, the scaling technique reduces the chances of an ill-conditioning of the matrix. This article proposes a one-parameter scaling memoryless Davidon–Fletcher–Powell (DFP) algorithm for solving a system of monotone nonlinear equations with convex constraints. The measure function that involves all the eigenvalues of the memoryless DFP matrix is minimized to obtain the scaling parameter’s optimal value. The resulting algorithm is matrix and derivative-free with low memory requirements and is globally convergent under some mild conditions. A numerical comparison showed that the algorithm is efficient in terms of the number of iterations, function evaluations, and CPU time. The performance of the algorithm is further illustrated by solving problems arising from image restoration.

A Hybrid Iterative Thresholding Method for Solving Sparsity-Regularized Linear Inverse Problems and Its Application in Seismic Sparse-Spike Deconvolution

Sparsity-regularized linear inverse problem has been served as the base in many disciplines, such as remote sensing imaging, image processing and analysis, seismic deconvolution, compressed sensing, medical imaging, and so forth. Iterative hard thresholding algorithm (IHTA) and iterative soft thresholding algorithm (ISTA) are two frequently used methods to solve sparsity-regularized linear inverse problems. They are also the basic unit of other more complex methods. IHTA and ISTA are derived under steepest descent method. I.e., iteratively perform gradient descent and thresholding shrinkage. Steepest descent method is a first order algorithm, which is a powerful way to solve optimization due to its relatively simple implementation. However, a known issue of first order method is possible poor convergence rate. Fast iterative thresholding-like algorithms have been proposed to overcome this issue in the existing literatures. In history, another alternative way is second order algorithms or quasi-second order algorithms. In this paper, we include a quasi-Newton’s method, i.e., DFP (Davidon-Fletcher-Powell) formulations in the framework of iterative thresholding-like algorithms to replace gradient descent to form a hybrid method to further increase convergence rate. The proposed method has been performed on two numerical examples and a real-life application in sparse-spike seismic deconvolution. The numerical examples and real-life application showed that it provides an effective alternative method to solve sparsity-regularized linear inverse problems.

Diagonally scaled memoryless quasi–Newton methods with application to compressed sensing

<p style='text-indent:20px;'>Memoryless quasi–Newton updating formulas of BFGS (Broyden–Fletcher–Goldfarb–Shanno) and DFP (Davidon–Fletcher–Powell) are scaled using well-structured diagonal matrices. In the scaling approach, diagonal elements as well as eigenvalues of the scaled memoryless quasi–Newton updating formulas play significant roles. Convergence analysis of the given diagonally scaled quasi–Newton methods is discussed. At last, performance of the methods is numerically tested on a set of CUTEr problems as well as the compressed sensing problem.</p>

A comparison of Quasi-Newton methods considering line search conditions in unconstrained minimization

Optimization algorithms are essential tools in all fields such as engineering, business, science and even in nature. Their performance and efficiency may differ from problem-to-problem. In this respect, the current paper aims to provide an understanding of the three well-known Quasi-Newton methods relation with various line search conditions on a performance basis. To this end, the fifteen Quasi-Newton method-line search condition combinations, which are composed of Symmetric-Rank-1 (SR-1), Broyden-Fletcher-Goldfarb-Shanno (BFGS) and Davidon-Fletcher-Powell (DFP) Quasi-Newton methods and Backtracking (BC), Armijo-Backtracking (ABC), Goldstein (GC), Weak Wolfe (WWC) and Strong Wolfe (SWC) conditions, are subjected to totally 195 computational experiments using thirteen test functions including smooth and non-smooth ones. During the experiments, the number of function evaluations at every iteration for all the combinations are kept track and the total number of function evaluations, when each combination converges, are set as a first performance metric. In addition to that, the eigenvalues of Hessian approximation matrix are checked if the updated matrix is positive definite or not. In case of having a negative definite Hessian approximation, the eigenvalue modification procedure is employed to guarantee that the updated matrix is positive definite, which in turn progress along the descent direction. Besides, for purpose of using in the performance evaluations, the number of negative definite matrix generations, as a second performance metric, are recorded for all the combinations. To conduct a reliable and an efficient performance analysis on the combinations, the performance and data profiles are used along with the number of negative definite Hessian matrix update generations. Based on the mathematical analysis through those data, the BFGS-GC combination is the fastest one whereas the slowest one is the DFP-SWC combination. For an optimal choice, it is determined that the BFGS-WWC combination is a great candidate. It is also observed that the BFGS and DFP methods operate very well as long as the curvature condition is satisfied by any line search conditions. However, this statement is not valid for the SR-1 method.

Non-asymptotic superlinear convergence of standard quasi-Newton methods

In this paper, we study and prove the non-asymptotic superlinear convergence rate of the Broyden class of quasi-Newton algorithms which includes the Davidon–Fletcher–Powell (DFP) method and the Broyden–Fletcher–Goldfarb–Shanno (BFGS) method. The asymptotic superlinear convergence rate of these quasi-Newton methods has been extensively studied in the literature, but their explicit finite–time local convergence rate is not fully investigated. In this paper, we provide a finite–time (non-asymptotic) convergence analysis for Broyden quasi-Newton algorithms under the assumptions that the objective function is strongly convex, its gradient is Lipschitz continuous, and its Hessian is Lipschitz continuous at the optimal solution. We show that in a local neighborhood of the optimal solution, the iterates generated by both DFP and BFGS converge to the optimal solution at a superlinear rate of (1/k)^{k/2}, where k is the number of iterations. We also prove a similar local superlinear convergence result holds for the case that the objective function is self-concordant. Numerical experiments on several datasets confirm our explicit convergence rate bounds. Our theoretical guarantee is one of the first results that provide a non-asymptotic superlinear convergence rate for quasi-Newton methods.

A New Generalized Quasi-Newton Algorithm Based on Structured Diagonal Hessian Approximation for Solving Nonlinear Least-Squares Problems With Application to 3DOF Planar Robot Arm Manipulator

Many problems in science and engineering can be formulated as nonlinear least-squares (NLS) problems. Thus, the need for efficient algorithms to solve these problems can not be overemphasized. In that sense, we introduce a generalized structured-based diagonal Hessian algorithm for solving NLS problems. The formulation associated with this algorithm is essentially a generalization of a similar result in Yahaya <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">et al.</i> (Journal of Computational and Applied Mathematics, pp. 113582, 2021). However, in this work, the structured diagonal Hessian update is derived under a weighted Frobenius norm; this allows other choices of the weighted matrix analogous to the Davidon-Fletcher-Powell (DFP) method. Moreover, to theoretically fill the gap in Yahaya <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">et al.</i> (Journal of Computational and Applied Mathematics, pp. 113582, 2021), we have shown that the proposed algorithm is R-linearly convergent under some standard conditions devoid of any safeguarding strategy. Furthermore, we experimentally tested the proposed scheme on some standard benchmark problems in the literature. Finally, we applied this algorithm to solve robotic motion control problem consisting of 3DOF (degrees of freedom).

Response EPSD of chain-like MDOF nonlinear structural systems via wavelet-Galerkin method

The periodic generalized harmonic wavelet (PGHW) method is used to analyze the response of the chain-like multi-degree-of-freedom (MDOF) nonlinear structural system with seismic excitation in the time and frequency domain. First, the theoretical background of PGHW is briefly introduced, and the relationship between the power spectral density (PSD) of the stochastic process and the corresponding wavelet coefficient is given. Next, the wavelet-Galerkin method is used to study the MDOF system, and a set of nonlinear algebraic equations can be obtained to get the wavelet coefficient of the response. The quasi-Newton method is selected to solve these equations. It is more efficient than Newton method because the Davidon-Fletcher-Powell (DFP) algorithm is used to approximate the Jacobian matrix in the iterative process. Then, the displacement and estimated power spectral density (EPSD) of the response can be obtained by using the solved wavelet coefficient. Finally, numerical examples of the single-degree-of-freedom (SDOF) and MDOF systems are shown to prove the feasibility and efficiency.

Hardware implementation of the DFP algorithm using inexact line search

The Quasi-Newton method (QN) is widely used in solving large-scale optimization problems due to its high efficiency. However, this algorithm requires a large number of iterative calculations, which is more time-consuming to compute in software. The Davidon-Fletcher-Powell (DFP) algorithm has been implemented on field-programmable gate array (FPGA) in order to speed up the computation. The number of iterations in the online search part of the algorithm is excessive, which affects the speed of the calculation. In order to solve this problem, this paper proposes a hardware implementation of DFP algorithm using inexact linear search. The results show that the calculation speed of the DFP algorithm implemented in this paper on the software side and FPGA platform is increased by 7.59 times and 3.08 times, respectively. And resource consumption and accuracy have little effect on the results.

Optimal decentralized control of a wind turbine and diesel generator system

AbstractThis article presents a decentralized optimal controller design technique for the frequency and power control of a coupled wind turbine and diesel generator. The decentralized controller consists of two proportional‐integral (PI)‐lead controllers which are designed and optimized simultaneously using a quasi‐Newton based optimization technique, namely, Davidon–Fletcher–Powell algorithm. The optimal PI‐lead controllers are designed in such a way that there are no communication links between them. Simulation results show the superior performance of the proposed controller with a lower order structure compared to the benchmark decentralized linear‐quadratic Gaussian integral controllers of orders 4 and 11. It is also shown that the proposed controller demonstrates an effective performance in damping the disturbances from load and wind power, as well as a robust performance against the parameter changes of the power system.

A PE-MUSIC Algorithm for Sparse Array in MIMO Radar

Larger array aperture is provided by sparse arrays than uniform ones, which can improve the angle estimation resolution and reduce the cost of system evidently. However, manifold ambiguity is introduced due to the array sparsity. In this paper, a Power Estimation Multiple-Signal Classification (PE-MUSIC) algorithm is proposed to solve the manifold ambiguity of arbitrary sparse arrays for uncorrelated sources in Multiple-Input Multiple-Output (MIMO) radar. First, the paired direction of departure (DOD) and direction of arrival (DOA) are obtained for all targets by MUSIC algorithm, including the true and spurious ones; then, the well-known Davidon–Fletcher–Powell (DFP) algorithm is applied to estimate all targets’ power values, among which the value of a spurious target trends to zero. Therefore, the ambiguity of sparse array in MIMO radar can be cleared. Simulation results verify the effectiveness and feasibility of the method.

Derivative-free method based on DFP updating formula for solving convex constrained nonlinear monotone equations and application

<abstract> In this paper, a new derivative-free approach for solving nonlinear monotone system of equations with convex constraints is proposed. The search direction of the proposed algorithm is derived based on the modified scaled Davidon-Fletcher-Powell (DFP) updating formula in such a way that it is sufficiently descent. Under some mild assumptions, the search direction is shown to be bounded. Subsequently, the convergence result of the proposed method is established. The performance of the proposed algorithm on a collection of some test problems as well as signal recovery problems is demonstrated in comparison with some existing algorithms with similar characteristics. The results of the numerical experiments confirm the efficiency as well as the robustness of the proposed algorithm by comparing it with some existing methods in the literature. </abstract>

High-Resolution ISAR Imaging with Modified Joint Range Spatial-Variant Autofocus and Azimuth Scaling.

Well-focused and accurately scaled high-resolution inverse synthetic aperture radar (ISAR) images provide a sound basis for feature extraction and target recognition. This paper proposes a novel high-resolution ISAR imaging algorithm, namely modified joint range spatial-variant autofocus and azimuth scaling algorithm (MJAAS). After motion compensation, the shift of the equivalent rotational center (ERC) of the target destroys the linear relationship between the azimuth chirp rates (ACR) of echo signals and the range coordinates of scattering points, thereby leading to the failure of azimuth scaling. Accordingly, a new joint equivalent rotational center position and effective rotational velocity (JERCP-ERV) signal model is established, serving as the basis of MJAAS. By recourse to the Davidon-Fletcher-Powell (DFP) algorithm, MJAAS can jointly estimate the ERCP and ERV by solving a minimum entropy optimization problem, so as to simultaneously achieve accurate azimuth scaling and range spatial-variant autofocus, which further improves the image focusing performance. MJAAS is not restricted by the modes of motion errors (coherent or non-coherent) and the motion compensation methods, so it can be widely applied to real data with the advantages of strong practicality and high accuracy. Extensive experimental results based on both simulated and real data are provided to corroborate the effectiveness of the proposed algorithm.

On the Derivation of Quasi-Newton Formulas for Optimization in Function Spaces

Newton’s method is usually preferred when solving optimization problems due to its superior convergence properties compared to gradient-based or derivative-free optimization algorithms. However, deriving and computing second-order derivatives needed by Newton’s method often is not trivial and, in some cases, not possible. In such cases quasi-Newton algorithms are a great alternative. In this paper, we provide a new derivation of well-known quasi-Newton formulas in an infinite-dimensional Hilbert space setting. It is known that quasi-Newton update formulas are solutions to certain variational problems over the space of symmetric matrices. In this paper, we formulate similar variational problems over the space of bounded symmetric operators in Hilbert spaces. By changing the constraints of the variational problem we obtain updates (for the Hessian and Hessian inverse) not only for the Broyden-Fletcher-Goldfarb-Shanno (BFGS) quasi-Newton method but also for Davidon–Fletcher–Powell (DFP), Symmetric Rank One (SR1), and Powell-Symmetric-Broyden (PSB). In addition, for an inverse problem governed by a partial differential equation (PDE), we derive DFP and BFGS “structured” secant formulas that explicitly use the derivative of the regularization and only approximates the second derivative of the misfit term. We show numerical results that demonstrate the desired mesh-independence property and superior performance of the resulting quasi-Newton methods.

Modified Conjugate Gradient Method via DFP Update for Solving Systems of Nonlinear Equations

In this study, a fully derivative-free method for solving large scale nonlinear systems of equations via memoryless DFP update is presented. The new proposed method is an enhanced DFP (Davidon-FletcherPowell) update which is matrix and derivative free thereby require low memory storage. Under suitable conditions, the proposed method converges globally. Numerical comparisons using a set of large-scale test problems showed that the proposed method is efficient.

A New Contactless Bubble/Slug Velocity Measurement Method of Gas–Liquid Two-Phase Flow in Small Channels

A new contactless bubble/slug velocity measurement method of the gas–liquid two-phase flow in small channels is proposed by extending the contactless conductivity detection technique to the contactless impedance detection (CID) technique. Two kinds of CID sensors with different constructions (tubular construction and radial construction) are developed to obtain the total impedance information (the real part, the imaginary part, and the amplitude) of the gas–liquid two-phase flow. Two bubble/slug velocity measurement systems with different constructions, the tubular three-electrode velocity measurement system and the radial four-electrode velocity measurement system, are developed. With the obtained total impedance information, three initial estimation values of the bubble/slug velocity are determined by the principle of the cross correlation velocity measurement. Then, with the real-time flow pattern identification result implemented by the fuzzy C-means, a corresponding velocity measurement model [which is the linear combination of the three initial estimation values of the bubble/slug velocity and is developed by the penalty function method and the Davidon–Fletcher–Powell algorithm] is applied to obtain the final bubble/slug velocity. Two groups of the bubble/slug velocity measurement system prototypes with three different inner diameters are developed. The bubble/slug velocity measurement experiments are carried out. The experimental results show that the proposed method is effective. For the two bubble/slug velocity measurement systems with different constructions, the maximum relative errors of the bubble/slug velocity measurement are all less than 4.5%. The research results also indicate that making full use of the total impedance information is an effective approach to improve the accuracy of the velocity measurement.