Unconstrained Minimization Problem Research Articles

An explicit spectral Fletcher–Reeves conjugate gradient method for bi-criteria optimization

Abstract In this paper, we propose a spectral Fletcher–Reeves conjugate gradient-like method for solving unconstrained bi-criteria minimization problems without using any technique of scalarization. We suggest an explicit formulae for computing a descent direction common to both criteria. The latter further verifies a sufficient descent property that does not depend on the line search nor on any convexity assumption. After proving the existence of a bi-criteria Armijo-type stepsize, global convergence of the proposed algorithm is established. Finally, some numerical results and comparisons with other methods are reported.

Network traffic recovery from link-load measurements using tensor triple decomposition strategy for third-order traffic tensors

Network traffic data is the pivot of input in many network tasks but its direct measurement can be insufferably costly. In this paper, we propose a network traffic recovery method which only requires the conveniently measurable link-load traffics. We arrange the traffic data as a third-order tensor and utilize the triple decomposition technique proposed very recently by Qi et al. (2021). The studied model is a differentialble unconstrained minimization problem, which can be efficiently solved by a Barzilai–Borwein (BB) gradient algorithm. We prove that the generated iteration sequence can globally converge to a certain stationary point of the objective function. The numerical simulations on three open-source traffic datasets demonstrate the superiority of our method in comparison with other state-of-the-art algorithms.

Minibatch Stochastic Three Points Method for Unconstrained Smooth Minimization

We present a new zero-order optimization method called Minibatch Stochastic Three Points (MiSTP), specifically designed to solve stochastic unconstrained minimization problems when only an approximate evaluation of the objective function is possible. MiSTP is an extension of the Stochastic Three Point Method (STP). The key innovation of MiSTP is that it selects the next point solely based on the objective function approximation, without relying on its exact evaluation. At each iteration, MiSTP generates a random search direction and compares the approximations of the objective function at the current point, the randomly generated direction and its opposite. The best of these three points is chosen as the next iterate. We analyze the worst-case complexity of MiSTP in the convex and non-convex cases and demonstrate that it matches the most accurate complexity bounds known in the literature for zero-order optimization methods. We perform extensive numerical evaluations to assess the computational efficiency of MiSTP and compare its performance to other state-of-the-art methods by testing it on several machine learning tasks. The results show that MiSTP outperforms or has comparable performance against state-of-the-art methods indicating its potential for a wide range of practical applications.

Two new limited-memory preconditioned conjugate gradient algorithms for nonlinear optimization problems

The conjugate gradient (CG) techniques are a class of unconstrained optimization algorithms with strong local and global convergence qualities and minimal memory needs. While the quasi-Newton methods are reliable and efficient on a wide range of problems and these methods are converge faster than the conjugate gradient methods and require fewer function evaluations, however, they are request substantially more storage, and if the problem is ill-conditioned, they may require several iterations. There is another class, termed preconditioned conjugate gradient method, it is a technique that combines two methods conjugate gradient with quasi-Newton. In this work, we proposed a new two limited memory preconditioned conjugate gradient methods (New1 and New2), to solve nonlinear unconstrained minimization problems, by using new modified symmetric rank one (NMSR1) and new modified Davidon, Fletcher, Powell (NMDFP), and also using projected vectors. We proved that these modifications fulfill some conditions. Also, the descent condition of the new technique has been proved. The numerical results showed the efficiency of the proposed new algorithms compared with some standard nonlinear, unconstrained problems.

An optimal choice Dai-Liao conjugate gradient algorithm for unconstrained optimization and portfolio selection

<abstract><p>In this research, we propose an optimal choice for the non-negative constant in the Dai-Liao conjugate gradient formula based on the prominent Barzilai-Borwein approach by leveraging the nice features of the Frobenius matrix norm. The global convergence of the new modification is demonstrated using some basic assumptions. Numerical comparisons with similar algorithms show that the new approach is reliable in terms of the number of iterations, computing time, and function evaluations for unconstrained minimization, portfolio selection and image restoration problems.</p></abstract>

Frequency-domain distributed multichannel wiener filtering speech enhancement algorithm

A frequency-domain distributed microphone multi-channel Wiener filter speech enhancement algorithm is proposed in this paper. In this paper, the distributed microphone speech model is considered. First, the speech signal in the time domain is converted into the speech signal in the frequency domain by the discrete Fourier transform method. Then, the unconstrained minimization problem of the noise reduction and speech distortion of the complex linear filter in the frequency domain is established. Simulation results show that the proposed algorithm is superior to some existing multi-channel speech enhancement algorithms.

Number of sites-based solver for determining coverages from steady-state mean-field micro-kinetic models.

Kinetic models parameterized by ab-initio calculations have led to significant improvements in understanding chemical reactions in heterogeneous catalysis. These studies have been facilitated by implementations which determine steady-state coverages and rates of mean-field micro-kinetic models. As implemented in the open-source kinetic modeling program, CatMAP, the conventional solution strategy is to use a root-finding algorithm to determine the coverage of all intermediates through the steady-state expressions, constraining all coverages to be non-negative and to properly sum to unity. Though intuitive, this root-finding strategy causes issues with convergence to solution due to these imposed constraints. In this work, we avoid explicitly imposing these constraints, solving the mean-field steady-state micro-kinetic model in the space of number of sites instead of solving it in the space of coverages. We transform the constrained root-finding problem to an unconstrained least-squares minimization problem, leading to significantly improved convergence in solving micro-kinetic models and thus enabling the efficient study of more complex catalytic reactions.

The Numerical Solution of Nonlinear Fractional Lienard and Duffing Equations Using Orthogonal Perceptron

This paper proposes an approximation algorithm based on the Legendre and Chebyshev artificial neural network to explore the approximate solution of fractional Lienard and Duffing equations with a Caputo fractional derivative. These equations show the oscillating circuit and generalize the spring–mass device equation. The proposed approach transforms the given nonlinear fractional differential equation (FDE) into an unconstrained minimization problem. The simulated annealing (SA) algorithm minimizes the mean square error. The proposed techniques examine various non-integer order problems to verify the theoretical results. The numerical results show that the proposed approach yields better results than existing methods.

Global convergence via modified self-adaptive approach for solving constrained monotone nonlinear equations with application to signal recovery problems

The conjugate gradient method (CG) is one of the most rapidly expanding and efficient ways of solving unconstrained minimization problems. Recently, there has been a lot of effort put into extending the CG approach to solve monotone nonlinear equations. In this paper, we describe a variation of the CG method for solving constrained monotone nonlinear equations. The approach has a sufficient descent property, and its global convergence has been demonstrated with the help of some reasonable assumptions. Two sets of numerical tests were run to demonstrate the proposed method’s superior performance when compared to other methods. The initial experiment aimed to solve nonlinear equations with constraints, while in the second experiment, the method was applied to sparse signal reconstruction.

SCORE: approximating curvature information under self-concordant regularization

Optimization problems that include regularization functions in their objectives are regularly solved in many applications. When one seeks second-order methods for such problems, it may be desirable to exploit specific properties of some of these regularization functions when accounting for curvature information in the solution steps to speed up convergence. In this paper, we propose the SCORE (self-concordant regularization) framework for unconstrained minimization problems which incorporates second-order information in the Newton-decrement framework for convex optimization. We propose the generalized Gauss–Newton with Self-Concordant Regularization (GGN-SCORE) algorithm that updates the minimization variables each time it receives a new input batch. The proposed algorithm exploits the structure of the second-order information in the Hessian matrix, thereby reducing computational overhead. GGN-SCORE demonstrates how to speed up convergence while also improving model generalization for problems that involve regularized minimization under the proposed SCORE framework. Numerical experiments show the efficiency of our method and its fast convergence, which compare favorably against baseline first-order and quasi-Newton methods. Additional experiments involving non-convex (overparameterized) neural network training problems show that the proposed method is promising for non-convex optimization.

Battery SOH estimation based on decision tree and improved support vector machine regression algorithm

Battery state of health (SOH) estimation is crucial for the estimation of the remaining driving range of electric vehicles and is one of the core functions of the battery management system (BMS). The lithium battery feature sample data used in this paper is extracted from the National Aeronautics and Space Administration (NASA) of the United States. Based on the obtained feature samples, a decision tree algorithm is used to analyze them and obtain the importance of each feature. Five groups of different feature inputs are constructed based on the cumulative feature importance, and the original support vector machine regression (SVR) algorithm is applied to perform SOH estimation simulation experiments on each group. The experimental results show that four battery features (voltage at SOC = 100%, voltage, discharge time, and SOC) can be used as input to achieve high estimation accuracy. To improve the training efficiency of the original SVR algorithm, an improved SVR algorithm is proposed, which optimizes the differentiability and solution method of the original SVR objective function. Since the loss function of the original SVR is non-differentiable, a smoothing function is introduced to approximate the loss function of the original SVR, and the original quadratic programming problem is transformed into a convex unconstrained minimization problem. The conjugate gradient algorithm is used to solve the smooth approximation objective function in a sequential minimal optimization manner. The improved SVR algorithm is applied to the simulation experiment with four battery feature inputs. The results show that the improved SVR algorithm significantly reduces the training time compared to the original SVR, with a slight trade-off in simulation accuracy.

Education

The Education section in this issue presents two contributions. In `"Nesterov's Method for Convex Optimization," Noel J. Walkington proposes a teaching guide for a first course in optimization of this well-known algorithm for computing the minimum of a convex function. This algorithm, first proposed in 1983 by Yuri Nesterov, though well recognized in computational optimization in the presence of large data as a more efficient tool than the steepest descent method, is still absent in most modern textbooks on optimization. The author of the present article develops an elementary analysis of Nesterov's first order algorithm that parallels that of steepest descent but with an additional requirement proposed by Nesterov. Two cases are discussed. The first concerns an unconstrained minimization problem, while the second includes closed convex constraints represented using infinite penalization of the cost. More generally, the cost function becomes the sum of a smooth convex function and a lower semicontinuous convex function. Several student-level exercises are included in this paper. Results are nicely illustrated by an example of a signal recovery problem and a discussion of the Uzawa algorithm for optimization problems with constraints defined by inequalities involving convex functions. The second paper, "A Comprehensive Proof of Bertrand's Theorem," is presented by Patrick De Leenheer, John Musgrove, and Tyler Schimleck. It concerns the behavior of the solutions of the classical two-body problem and states that, among all possible gravitational laws, there are only two exhibiting the property that all bounded orbits are closed: Newtonian gravitation and Hookean gravitation. Historically, even if Newton was aware that there are to specific gravitational laws having the above property, it was only two centuries later, in 1873, that Bertrand realized that these are the only ones. Bertrand's theorem, due to its important consequences, has been integrated into the undergraduate curriculum in theoretical mechanics, but its proof, accessible to undergraduate mathematics or physics students, seems to be absent from modern textbooks. Although Bertrand's original paper did not contain a precise proof, V. Arnold proposed a sketch of it based on six subproblems. Among other contributions, this article provides a complete proof of the sixth subproblem under a specific assumption imposed on the magnitude of the force in the motion model. Under this assumption, a complete proof of Bertrand's theorem is then given, incorporating also earlier contributions by other authors. Still, comprehensive does not mean simple here, and this paper may be used to conceive several research projects for advanced-level undergraduate students in mathematics or physics.

Multiplier waveform inversion: A reduced-space full-waveform inversion by the method of multipliers

Full-waveform inversion (FWI) is a nonlinear optimization problem that addresses the estimation of subsurface model parameters by matching the predicted to the observed seismograms. We formulate FWI as a constrained optimization problem in which the regularization term is minimized subject to the nonlinear data matching constraints. Unlike standard FWI, which solves this regularized problem with a penalty method, we use an augmented Lagrangian formulation in the framework of the method of multipliers (MM). This leads to a two-step recursive algorithm, which is called multiplier waveform inversion (MWI). First, the primal step solves an unconstrained minimization problem, which is similar to the standard FWI but with a modified data term. These modified data are obtained by adding the Lagrange multipliers to the data residuals of the classical FWI. Then, the dual step updates the Lagrange multipliers with basic gradient ascent steps. In this framework, they reduce to the running sum of the data residuals of the previous iterations. The performance of the overall algorithm is improved by considering that the MM does not require the exact solution of the primal step at each iteration. In fact, convergence is attained when the primal subproblem is solved with one (without an inner loop) iteration of a gradient-based method at each iteration of the MWI. The new algorithm can be easily implemented in existing FWI codes without computational overhead by adding the Lagrange multipliers to the data residuals at each iteration. We find with numerical examples that MWI has a faster convergence speed and much improved stability compared with FWI and can converge to an accurate solution of the inverse problem in the absence of low-frequency data, even with a constant step size.

A new fuzzy support vector machine with pinball loss

The fuzzy support vector machine (FSVM) assigns each sample a fuzzy membership value based on its relevance, making it less sensitive to noise or outliers in the data. Although FSVM has had some success in avoiding the negative effects of noise, it uses hinge loss, which maximizes the shortest distance between two classes and is ineffective in dealing with feature noise near the decision boundary. Furthermore, whereas FSVM concentrates on misclassification errors, it neglects to consider the critical within-class scatter minimization. We present a Fuzzy support vector machine with pinball loss (FPin-SVM), which is a fuzzy extension of a reformulation of a recently proposed support vector machine with pinball loss (Pin-SVM) with several significant improvements, to improve the performance of FSVM. First, because we used the squared L2- norm of errors variables instead of the L1 norm, our FPin-SVM is a strongly convex minimization problem; second, to speed up the training procedure, solutions of the proposed FPin-SVM, as an unconstrained minimization problem, are obtained using the functional iterative and Newton methods. Third, it is proposed to solve the minimization problem directly in primal. Unlike FSVM and Pin-SVM, our FPin-SVM does not require a toolbox for optimization. We dig deeper into the features of FPin-SVM, such as noise insensitivity and within-class scatter minimization. We conducted experiments on synthetic and real-world datasets with various sounds to validate the usefulness of the suggested approach. Compared to the SVM, FSVM, and Pin-SVM, the presented approaches demonstrate equivalent or superior generalization performance in less training time.

Derivative-free separable quadratic modeling and cubic regularization for unconstrained optimization

We present a derivative-free separable quadratic modeling and cubic regularization technique for solving smooth unconstrained minimization problems. The derivative-free approach is mainly concerned with building a quadratic model that could be generated by numerical interpolation or using a minimum Frobenius norm approach, when the number of points available does not allow to build a complete quadratic model. This model plays a key role to generate an approximated gradient vector and Hessian matrix of the objective function at every iteration. We add a specialized cubic regularization strategy to minimize the quadratic model at each iteration, that makes use of separability. We discuss convergence results, including worst case complexity, of the proposed schemes to first-order stationary points. Some preliminary numerical results are presented to illustrate the robustness of the specialized separable cubic algorithm.

Efficient linear reformulations for binary polynomial optimization problems

We consider unconstrained polynomial minimization problems with binary variables (BPO). These problems can be easily linearized, i.e., reformulated into a MILP in a higher dimensional space. Several linearizations are possible for a given BPO, depending on how each monomial is decomposed and replaced by additional variables and constraints. We focus on finding efficient linearizations that maximize the continuous relaxation bound of the resulting MILP. For this purpose, we introduce the notion of linearization patterns that allow us to model and enumerate the possible decompositions of a degree-d monomial. The assignment of a unique pattern to each monomial of BPO results in a reformulation of BPO into a MILP. Our method, called MaxBound, amounts to searching for an optimal association between monomials and patterns in the sense that it leads to a MILP with the best continuous relaxation bound. We show that this process can be formulated as a MILP which we denote by (MB˜). We further highlight domination properties among the patterns that allow us to discard the dominated patterns and to decrease the size of (MB˜). Another effect of these domination properties is that it now makes sense to search for a reformulation that requires as few additional variables as possible, based only on the non-dominated patterns. We call this reformulation method ND-MinVar and again show that it can be found by solving another MILP. We make an experimental study on degree 4 polynomials that compares the results of both methods and shows the advantages and disadvantages of each.

A robust stress update algorithm for elastoplastic models without analytical derivation of the consistent tangent operator and loading/unloading estimation

AbstractA robust and concise implicit stress integration algorithm of elastoplastic models is presented. It does not require the loading/unloading estimation and analytical derivation operation for the stress update. First, the elastoplastic stress update problem is recast into an unconstrained minimization problem by utilizing the smooth function to bypass the loading/unloading estimation. Then, the object problem is solved by the line search method instead of the Newton method for better convergence. The consistent tangent operator is evaluated by the complex step derivative approximation without the subtraction cancellation error, which provides the quadratic convergence rate of global iteration. The rationality of the numerical consistent tangent operator is validated by the one obtained by the analytical derivation. A recently developed non‐orthogonal elastoplastic (NEP) clay model is implemented using the new algorithm. The algorithm is confirmed through comparing the numerical solution and the analytical one for a cavity expansion problem. The algorithm performance is assessed based on a series of geotechnical boundary value problems. It is found that the new algorithm is more robust than the one employed by ABAQUS. The source code of the model implementation can be downloaded from https://github.com/zhouxin615.

A dynamic model to solve weighted linear complementarity problems

A dynamic model (DM) is proposed to solve the weighted linear complementarity problem (WLCP). The DM is based on an approximate reformulation of the WLCP into an unconstrained minimization problem. Under certain hypothesis, we prove that the equilibrium point of the DM is asymptotically stable and it is an approximate solution of the monotone WLCP. Simulation results are given to show the effectiveness of our method.

A New Modified Conjugate Gradient for Nonlinear Minimization Problems

The conjugate gradient is a highly effective technique to solve the unconstrained nonlinear minimization problems and it is one of the most well-known methods. It has a lot of applications. For large-scale, unconstrained minimization problems, conjugate gradient techniques are widely applied. In this paper, we will suggest a new parameter of conjugate gradient to solve the nonlinear unconstrained minimization problems, based on parameter of Dai and Liao. We will study the property of the descent ,the property of the sufficient descent and property of the global convergence a new method. We introduce some numerical data to prove the efficacy of the our method.

An extended delayed weighted gradient algorithm for solving strongly convex optimization problems

The recently developed delayed weighted gradient method (DWGM) is competitive with the well-known conjugate gradient (CG) method for the minimization of strictly convex quadratic functions. As well as the CG method, DWGM has some key optimality and orthogonality properties that justify its practical performance. The main difference with the CG method is that, instead of minimizing the objective function on the entire explored subspace, DWGM minimizes the 2-norm of the gradient vector on the same subspace. The main purpose of this study is to extend DWGM for solving strongly convex nonquadratic minimization problems while keeping a low computational cost per iteration. We incorporate the scheme into a tolerant line search globalization strategy, and we show that it exhibits q-linear convergence to the unique global solution. We compare the proposed extended DWGM with state-of-the-art methods for large-scale unconstrained minimization problems. We use some well-known strongly convex test problems, but also solve some regularized logistic regression problems that appear in machine learning. Our numerical results illustrate that the proposed scheme is promising and exhibits a fast convergence behavior. Moreover, we show through numerical experiments on CUTEst problems that the proposed extended DWGM can be very effective in accelerating the convergence of a well-established Barzilai–Borwein-type method when the iterates get close to minimizers of non-convex functions.