Unconstrained Minimization Problem Research Articles

A Neural Network Approach for Solving Weighted Nonlinear Complementarity Problems

In this paper, we consider the weighted nonlinear complementarity problem which is the extension of the general complementarity problems and contains a wide class of optimization problems. Based on the weighted complementarity function, we reformulate the weighted nonlinear complementarity problem as an unconstrained minimization problem and present the steepest descent-based neural network to solve it. Under mild conditions, we derive some results related to the asymptotic stability of the neural network. Numerical experiments indicate that the proposed algorithm is quite effective.

An efficient angle-based twin random vector functional link classifier

Random vector functional link (RVFL) has always proven to be an excellent classifier in various application areas of machine learning. In this work, inspired by RVFL and its twin variant, i.e., twin RVFL (TRVFL), we propose an efficient angle-based twin random vector functional link (ATRVFL) to address the binary classification problem. ATRVFL is a general classification model in which the first optimization problem can be found by solving a quadratic programming problem (QPP). The second problem can be described as an unconstrained minimization problem (UMP) which is solved by a system of linear equations. ATRVFL finds the solution by solving a QPP and a UMP instead of solving two QPPs. Therefore, its training time is significantly less than TRVFL. To maximize the angle between the normals of the two hyperplanes, the second hyperplane is chosen to be close to its class. The objective of introducing the idea of angle is to maximise the distance between two hyperplanes to improve the ability to discriminate them in geometric spaces. Experimental analyses are performed on a leaf dataset and 25 real-world benchmark datasets (RWDs) collected from the UCI repository of various fields like medical, biological, etc. The results are evaluated based on classification accuracy, the area under the curve, G-mean and F1-score. The results of ATRVFL are compared with Support Vector Machine (SVM), RVFL, Twin-RVFL (TRVFL), RVFL with ε-insensitive Huber loss (ε-HRVFL), ensemble deep learning-based RVFL network (edRVFL) and Intuitionistic fuzzy random vector functional link classifier (IFRVFLC). The classification accuracy of the proposed model on the leaf dataset is 91.093 % and the highest F1-score of 0.768. Overall, based on experimental analysis it is evident that the proposed ATRVFL shows comparative or better performance than SVM, RVFL, TRVFL, ε-HRVFL, IFRVFLC and edRVFL.

Generalized div-curl based regularization for physically constrained deformable image registration

Variational image registration methods commonly employ a similarity metric and a regularization term that renders the minimization problem well-posed. However, many frequently used regularizations such as smoothness or curvature do not necessarily reflect the underlying physics that apply to anatomical deformations. This, in turn, can make the accurate estimation of complex deformations particularly challenging. Here, we present a new highly flexible regularization inspired from the physics of fluid dynamics which allows applying independent penalties on the divergence and curl of the deformations and/or their nth order derivative. The complexity of the proposed generalized div-curl regularization renders the problem particularly challenging using conventional optimization techniques. To this end, we develop a transformation model and an optimization scheme that uses the divergence and curl components of the deformation as control parameters for the registration. We demonstrate that the original unconstrained minimization problem reduces to a constrained problem for which we propose the use of the augmented Lagrangian method. Doing this, the equations of motion greatly simplify and become managable. Our experiments indicate that the proposed framework can be applied on a variety of different registration problems and produce highly accurate deformations with the desired physical properties.

A Descent Matrix-free Nonlinear Conjugate Gradient Algorithm for Impulse Noise Removal

The convergence of the Polak, Ribie‘re-Polyak (PRP) conjugate gradient (CG) method requires some modifications for improved theoretical properties. In this article, we explore an optimal choice of the Perry conjugacy condition to propose a hybrid CG parameter for solving optimization and inverse problems, particularly in an image reconstruction model. This parameter is selected to satisfy a combination of revised version of the PRP and Dai-Yuan (DY) CG methods. The numerical implementation includes inexact line search, showcasing the scheme's robustness (highest number of solved functions) compared to other known CG algorithms. The efficiency is shown in terms of Real error (RelErr), peak signal noise ratio (PNSR), and CPU time in seconds for impulse noise removal while for unconstrained minimization problems, the study evaluated the efficiency based on number of iterations, function evaluation, and CPU time in seconds. An interesting feature of the proposed method is its ability to converges to the minimizer regardless of the initial guess, relying on certain established assumptions.

Exploring Enhanced Conjugate Gradient Methods: A Novel Family of Techniques for Efficient Unconstrained Minimization

Given that the conjugate gradient method (CGM) is computationally efficient and user-friendly, it is often used to address large-scale, unconstrained minimization issues. Numerous researchers have created new conjugate gradient (CG) update parameters by modifying the initial set, also referred to as classical CGMs. This has resulted in the development of several hybrid approaches. This work's major goal is to create a new family of techniques that can be used to create even more new methods. Consequently, Hestenes-Stiefel's update parameter and a new family involving Polak-Ribiere-Polyak and Liu-Storey CGMs are considered. By changing the parameters of this CGM family, a novel approach that possesses sufficient descent characteristics is obtained. A numerical experiment including many unconstrained minimization problems (UMP) is carried out to assess the novel method's efficacy compared to existing approaches. The result reveals that the new CG approach performs better than the current ones.

An explicit spectral Fletcher–Reeves conjugate gradient method for bi-criteria optimization

Abstract In this paper, we propose a spectral Fletcher–Reeves conjugate gradient-like method for solving unconstrained bi-criteria minimization problems without using any technique of scalarization. We suggest an explicit formulae for computing a descent direction common to both criteria. The latter further verifies a sufficient descent property that does not depend on the line search nor on any convexity assumption. After proving the existence of a bi-criteria Armijo-type stepsize, global convergence of the proposed algorithm is established. Finally, some numerical results and comparisons with other methods are reported.

Network traffic recovery from link-load measurements using tensor triple decomposition strategy for third-order traffic tensors

Network traffic data is the pivot of input in many network tasks but its direct measurement can be insufferably costly. In this paper, we propose a network traffic recovery method which only requires the conveniently measurable link-load traffics. We arrange the traffic data as a third-order tensor and utilize the triple decomposition technique proposed very recently by Qi et al. (2021). The studied model is a differentialble unconstrained minimization problem, which can be efficiently solved by a Barzilai–Borwein (BB) gradient algorithm. We prove that the generated iteration sequence can globally converge to a certain stationary point of the objective function. The numerical simulations on three open-source traffic datasets demonstrate the superiority of our method in comparison with other state-of-the-art algorithms.

Minibatch Stochastic Three Points Method for Unconstrained Smooth Minimization

We present a new zero-order optimization method called Minibatch Stochastic Three Points (MiSTP), specifically designed to solve stochastic unconstrained minimization problems when only an approximate evaluation of the objective function is possible. MiSTP is an extension of the Stochastic Three Point Method (STP). The key innovation of MiSTP is that it selects the next point solely based on the objective function approximation, without relying on its exact evaluation. At each iteration, MiSTP generates a random search direction and compares the approximations of the objective function at the current point, the randomly generated direction and its opposite. The best of these three points is chosen as the next iterate. We analyze the worst-case complexity of MiSTP in the convex and non-convex cases and demonstrate that it matches the most accurate complexity bounds known in the literature for zero-order optimization methods. We perform extensive numerical evaluations to assess the computational efficiency of MiSTP and compare its performance to other state-of-the-art methods by testing it on several machine learning tasks. The results show that MiSTP outperforms or has comparable performance against state-of-the-art methods indicating its potential for a wide range of practical applications.

Two new limited-memory preconditioned conjugate gradient algorithms for nonlinear optimization problems

The conjugate gradient (CG) techniques are a class of unconstrained optimization algorithms with strong local and global convergence qualities and minimal memory needs. While the quasi-Newton methods are reliable and efficient on a wide range of problems and these methods are converge faster than the conjugate gradient methods and require fewer function evaluations, however, they are request substantially more storage, and if the problem is ill-conditioned, they may require several iterations. There is another class, termed preconditioned conjugate gradient method, it is a technique that combines two methods conjugate gradient with quasi-Newton. In this work, we proposed a new two limited memory preconditioned conjugate gradient methods (New1 and New2), to solve nonlinear unconstrained minimization problems, by using new modified symmetric rank one (NMSR1) and new modified Davidon, Fletcher, Powell (NMDFP), and also using projected vectors. We proved that these modifications fulfill some conditions. Also, the descent condition of the new technique has been proved. The numerical results showed the efficiency of the proposed new algorithms compared with some standard nonlinear, unconstrained problems.

An optimal choice Dai-Liao conjugate gradient algorithm for unconstrained optimization and portfolio selection

<abstract><p>In this research, we propose an optimal choice for the non-negative constant in the Dai-Liao conjugate gradient formula based on the prominent Barzilai-Borwein approach by leveraging the nice features of the Frobenius matrix norm. The global convergence of the new modification is demonstrated using some basic assumptions. Numerical comparisons with similar algorithms show that the new approach is reliable in terms of the number of iterations, computing time, and function evaluations for unconstrained minimization, portfolio selection and image restoration problems.</p></abstract>

Frequency-domain distributed multichannel wiener filtering speech enhancement algorithm

A frequency-domain distributed microphone multi-channel Wiener filter speech enhancement algorithm is proposed in this paper. In this paper, the distributed microphone speech model is considered. First, the speech signal in the time domain is converted into the speech signal in the frequency domain by the discrete Fourier transform method. Then, the unconstrained minimization problem of the noise reduction and speech distortion of the complex linear filter in the frequency domain is established. Simulation results show that the proposed algorithm is superior to some existing multi-channel speech enhancement algorithms.

Number of sites-based solver for determining coverages from steady-state mean-field micro-kinetic models.

Kinetic models parameterized by ab-initio calculations have led to significant improvements in understanding chemical reactions in heterogeneous catalysis. These studies have been facilitated by implementations which determine steady-state coverages and rates of mean-field micro-kinetic models. As implemented in the open-source kinetic modeling program, CatMAP, the conventional solution strategy is to use a root-finding algorithm to determine the coverage of all intermediates through the steady-state expressions, constraining all coverages to be non-negative and to properly sum to unity. Though intuitive, this root-finding strategy causes issues with convergence to solution due to these imposed constraints. In this work, we avoid explicitly imposing these constraints, solving the mean-field steady-state micro-kinetic model in the space of number of sites instead of solving it in the space of coverages. We transform the constrained root-finding problem to an unconstrained least-squares minimization problem, leading to significantly improved convergence in solving micro-kinetic models and thus enabling the efficient study of more complex catalytic reactions.

The Numerical Solution of Nonlinear Fractional Lienard and Duffing Equations Using Orthogonal Perceptron

This paper proposes an approximation algorithm based on the Legendre and Chebyshev artificial neural network to explore the approximate solution of fractional Lienard and Duffing equations with a Caputo fractional derivative. These equations show the oscillating circuit and generalize the spring–mass device equation. The proposed approach transforms the given nonlinear fractional differential equation (FDE) into an unconstrained minimization problem. The simulated annealing (SA) algorithm minimizes the mean square error. The proposed techniques examine various non-integer order problems to verify the theoretical results. The numerical results show that the proposed approach yields better results than existing methods.

Global convergence via modified self-adaptive approach for solving constrained monotone nonlinear equations with application to signal recovery problems

The conjugate gradient method (CG) is one of the most rapidly expanding and efficient ways of solving unconstrained minimization problems. Recently, there has been a lot of effort put into extending the CG approach to solve monotone nonlinear equations. In this paper, we describe a variation of the CG method for solving constrained monotone nonlinear equations. The approach has a sufficient descent property, and its global convergence has been demonstrated with the help of some reasonable assumptions. Two sets of numerical tests were run to demonstrate the proposed method’s superior performance when compared to other methods. The initial experiment aimed to solve nonlinear equations with constraints, while in the second experiment, the method was applied to sparse signal reconstruction.

SCORE: approximating curvature information under self-concordant regularization

Optimization problems that include regularization functions in their objectives are regularly solved in many applications. When one seeks second-order methods for such problems, it may be desirable to exploit specific properties of some of these regularization functions when accounting for curvature information in the solution steps to speed up convergence. In this paper, we propose the SCORE (self-concordant regularization) framework for unconstrained minimization problems which incorporates second-order information in the Newton-decrement framework for convex optimization. We propose the generalized Gauss–Newton with Self-Concordant Regularization (GGN-SCORE) algorithm that updates the minimization variables each time it receives a new input batch. The proposed algorithm exploits the structure of the second-order information in the Hessian matrix, thereby reducing computational overhead. GGN-SCORE demonstrates how to speed up convergence while also improving model generalization for problems that involve regularized minimization under the proposed SCORE framework. Numerical experiments show the efficiency of our method and its fast convergence, which compare favorably against baseline first-order and quasi-Newton methods. Additional experiments involving non-convex (overparameterized) neural network training problems show that the proposed method is promising for non-convex optimization.

Battery SOH estimation based on decision tree and improved support vector machine regression algorithm

Battery state of health (SOH) estimation is crucial for the estimation of the remaining driving range of electric vehicles and is one of the core functions of the battery management system (BMS). The lithium battery feature sample data used in this paper is extracted from the National Aeronautics and Space Administration (NASA) of the United States. Based on the obtained feature samples, a decision tree algorithm is used to analyze them and obtain the importance of each feature. Five groups of different feature inputs are constructed based on the cumulative feature importance, and the original support vector machine regression (SVR) algorithm is applied to perform SOH estimation simulation experiments on each group. The experimental results show that four battery features (voltage at SOC = 100%, voltage, discharge time, and SOC) can be used as input to achieve high estimation accuracy. To improve the training efficiency of the original SVR algorithm, an improved SVR algorithm is proposed, which optimizes the differentiability and solution method of the original SVR objective function. Since the loss function of the original SVR is non-differentiable, a smoothing function is introduced to approximate the loss function of the original SVR, and the original quadratic programming problem is transformed into a convex unconstrained minimization problem. The conjugate gradient algorithm is used to solve the smooth approximation objective function in a sequential minimal optimization manner. The improved SVR algorithm is applied to the simulation experiment with four battery feature inputs. The results show that the improved SVR algorithm significantly reduces the training time compared to the original SVR, with a slight trade-off in simulation accuracy.

Multiplier waveform inversion: A reduced-space full-waveform inversion by the method of multipliers

Full-waveform inversion (FWI) is a nonlinear optimization problem that addresses the estimation of subsurface model parameters by matching the predicted to the observed seismograms. We formulate FWI as a constrained optimization problem in which the regularization term is minimized subject to the nonlinear data matching constraints. Unlike standard FWI, which solves this regularized problem with a penalty method, we use an augmented Lagrangian formulation in the framework of the method of multipliers (MM). This leads to a two-step recursive algorithm, which is called multiplier waveform inversion (MWI). First, the primal step solves an unconstrained minimization problem, which is similar to the standard FWI but with a modified data term. These modified data are obtained by adding the Lagrange multipliers to the data residuals of the classical FWI. Then, the dual step updates the Lagrange multipliers with basic gradient ascent steps. In this framework, they reduce to the running sum of the data residuals of the previous iterations. The performance of the overall algorithm is improved by considering that the MM does not require the exact solution of the primal step at each iteration. In fact, convergence is attained when the primal subproblem is solved with one (without an inner loop) iteration of a gradient-based method at each iteration of the MWI. The new algorithm can be easily implemented in existing FWI codes without computational overhead by adding the Lagrange multipliers to the data residuals at each iteration. We find with numerical examples that MWI has a faster convergence speed and much improved stability compared with FWI and can converge to an accurate solution of the inverse problem in the absence of low-frequency data, even with a constant step size.

Education

The Education section in this issue presents two contributions. In `"Nesterov's Method for Convex Optimization," Noel J. Walkington proposes a teaching guide for a first course in optimization of this well-known algorithm for computing the minimum of a convex function. This algorithm, first proposed in 1983 by Yuri Nesterov, though well recognized in computational optimization in the presence of large data as a more efficient tool than the steepest descent method, is still absent in most modern textbooks on optimization. The author of the present article develops an elementary analysis of Nesterov's first order algorithm that parallels that of steepest descent but with an additional requirement proposed by Nesterov. Two cases are discussed. The first concerns an unconstrained minimization problem, while the second includes closed convex constraints represented using infinite penalization of the cost. More generally, the cost function becomes the sum of a smooth convex function and a lower semicontinuous convex function. Several student-level exercises are included in this paper. Results are nicely illustrated by an example of a signal recovery problem and a discussion of the Uzawa algorithm for optimization problems with constraints defined by inequalities involving convex functions. The second paper, "A Comprehensive Proof of Bertrand's Theorem," is presented by Patrick De Leenheer, John Musgrove, and Tyler Schimleck. It concerns the behavior of the solutions of the classical two-body problem and states that, among all possible gravitational laws, there are only two exhibiting the property that all bounded orbits are closed: Newtonian gravitation and Hookean gravitation. Historically, even if Newton was aware that there are to specific gravitational laws having the above property, it was only two centuries later, in 1873, that Bertrand realized that these are the only ones. Bertrand's theorem, due to its important consequences, has been integrated into the undergraduate curriculum in theoretical mechanics, but its proof, accessible to undergraduate mathematics or physics students, seems to be absent from modern textbooks. Although Bertrand's original paper did not contain a precise proof, V. Arnold proposed a sketch of it based on six subproblems. Among other contributions, this article provides a complete proof of the sixth subproblem under a specific assumption imposed on the magnitude of the force in the motion model. Under this assumption, a complete proof of Bertrand's theorem is then given, incorporating also earlier contributions by other authors. Still, comprehensive does not mean simple here, and this paper may be used to conceive several research projects for advanced-level undergraduate students in mathematics or physics.

A new fuzzy support vector machine with pinball loss

The fuzzy support vector machine (FSVM) assigns each sample a fuzzy membership value based on its relevance, making it less sensitive to noise or outliers in the data. Although FSVM has had some success in avoiding the negative effects of noise, it uses hinge loss, which maximizes the shortest distance between two classes and is ineffective in dealing with feature noise near the decision boundary. Furthermore, whereas FSVM concentrates on misclassification errors, it neglects to consider the critical within-class scatter minimization. We present a Fuzzy support vector machine with pinball loss (FPin-SVM), which is a fuzzy extension of a reformulation of a recently proposed support vector machine with pinball loss (Pin-SVM) with several significant improvements, to improve the performance of FSVM. First, because we used the squared L2- norm of errors variables instead of the L1 norm, our FPin-SVM is a strongly convex minimization problem; second, to speed up the training procedure, solutions of the proposed FPin-SVM, as an unconstrained minimization problem, are obtained using the functional iterative and Newton methods. Third, it is proposed to solve the minimization problem directly in primal. Unlike FSVM and Pin-SVM, our FPin-SVM does not require a toolbox for optimization. We dig deeper into the features of FPin-SVM, such as noise insensitivity and within-class scatter minimization. We conducted experiments on synthetic and real-world datasets with various sounds to validate the usefulness of the suggested approach. Compared to the SVM, FSVM, and Pin-SVM, the presented approaches demonstrate equivalent or superior generalization performance in less training time.

Derivative-free separable quadratic modeling and cubic regularization for unconstrained optimization

We present a derivative-free separable quadratic modeling and cubic regularization technique for solving smooth unconstrained minimization problems. The derivative-free approach is mainly concerned with building a quadratic model that could be generated by numerical interpolation or using a minimum Frobenius norm approach, when the number of points available does not allow to build a complete quadratic model. This model plays a key role to generate an approximated gradient vector and Hessian matrix of the objective function at every iteration. We add a specialized cubic regularization strategy to minimize the quadratic model at each iteration, that makes use of separability. We discuss convergence results, including worst case complexity, of the proposed schemes to first-order stationary points. Some preliminary numerical results are presented to illustrate the robustness of the specialized separable cubic algorithm.