Method For Unconstrained Minimization Research Articles

Minibatch Stochastic Three Points Method for Unconstrained Smooth Minimization

We present a new zero-order optimization method called Minibatch Stochastic Three Points (MiSTP), specifically designed to solve stochastic unconstrained minimization problems when only an approximate evaluation of the objective function is possible. MiSTP is an extension of the Stochastic Three Point Method (STP). The key innovation of MiSTP is that it selects the next point solely based on the objective function approximation, without relying on its exact evaluation. At each iteration, MiSTP generates a random search direction and compares the approximations of the objective function at the current point, the randomly generated direction and its opposite. The best of these three points is chosen as the next iterate. We analyze the worst-case complexity of MiSTP in the convex and non-convex cases and demonstrate that it matches the most accurate complexity bounds known in the literature for zero-order optimization methods. We perform extensive numerical evaluations to assess the computational efficiency of MiSTP and compare its performance to other state-of-the-art methods by testing it on several machine learning tasks. The results show that MiSTP outperforms or has comparable performance against state-of-the-art methods indicating its potential for a wide range of practical applications.

Global convergence of a nonmonotone Broyden family method for nonconvex unconstrained minimization

Global convergence of a nonmonotone Broyden family method for nonconvex unconstrained minimization

$$\rho$$-regularization subproblems: strong duality and an eigensolver-based algorithm

Trust-region (TR) type method, based on a quadratic model such as the trust-region subproblem (TRS) and p-regularization subproblem (pRS), is arguably one of the most successful methods for unconstrained minimization. In this paper, we study a general regularized subproblem (named \(\rho\)RS), which covers TRS and pRS as special cases. We derive a strong duality theorem for \(\rho\)RS, and also its necessary and sufficient optimality condition under general assumptions on the regularization term. We then define the Rendl–Wolkowicz (RW) dual problem of \(\rho\)RS, which is a maximization problem whose objective function is concave, and differentiable except possibly at two points. It is worth pointing out that our definition is based on an alternative derivation of the RW-dual problem for TRS. Then we propose an eigensolver-based algorithm for solving the RW-dual problem of \(\rho\)RS. The algorithm is carried out by finding the smallest eigenvalue and its unit eigenvector of a certain matrix in each iteration. Finally, we present numerical results on randomly generated pRS’s, and on a new class of regularized problem that combines TRS and pRS, to illustrate our algorithm.

Direct energy minimization based on exponential transformation in density functional calculations of finite and extended systems

The energy minimization involved in density functional calculations of electronic systems can be carried out using an exponential transformation that preserves the orthonormality of the orbitals. The energy of the system is then represented as a function of the elements of a skew-Hermitian matrix that can be optimized directly using unconstrained minimization methods. An implementation based on the limited memory Broyden-Fletcher-Goldfarb-Shanno approach with inexact line search and a preconditioner is presented and the performance compared with that of the commonly used self-consistent field approach. Results are presented for the G2 set of 148 molecules, liquid water configurations with up to 576 molecules and some insulating crystals. A general preconditioner is presented that is applicable to systems with fractional orbital occupation as is, for example, needed in the k-point sampling for periodic systems. This exponential transformation direct minimization approach is found to outperform the standard implementation of the self-consistent field approach in that all the calculations converge with the same set of parameter values and it requires less computational effort on average. The formulation of the exponential transformation and the gradients of the energy presented here are quite general and can be applied to energy functionals that are not unitary invariant such as self-interaction corrected functionals.

Joint Offloading and Energy Harvesting Design in Multiple Time Blocks for FDMA Based Wireless Powered MEC

The combination of mobile edge computing (MEC) and wireless power transfer (WPT) is recognized as a promising technology to solve the problem of limited battery capacities and insufficient computation capabilities of mobile devices. This technology can transfer energy to users by radio frequency (RF) in wireless powered mobile edge computing. The user converts the harvested energy, stores it in the battery, and utilizes the harvested energy to execute corresponding local computing and offloading tasks. This paper adopts the Frequency Division Multiple Access (FDMA) technique to achieve task offloading from multiple mobile devices to the MEC server simultaneously. Our objective is to study multiuser dynamic joint optimization of computation and wireless resource allocation under multiple time blocks to solve the problem of maximizing residual energy. To this end, we formalize it as a nonconvex problem that jointly optimizes the number of offloaded bits, energy harvesting time, and transmission bandwidth. We adopt convex optimization technology, combine with Karush–Kuhn–Tucker (KKT) conditions, and finally transform the problem into a univariate constrained convex optimization problem. Furthermore, to solve the problem, we propose a combined method of Bisection method and sequential unconstrained minimization based on Reformulation-Linearization Technique (RLT). Numerical results demonstrate that the performance of our joint optimization method outperforms other benchmark schemes for the residual energy maximization problem. Besides, the algorithm can maximize the residual energy, reduce the computation complexity, and improve computation efficiency.

Accelerated Meta-Algorithm for Convex Optimization Problems

We propose an accelerated meta-algorithm, which allows to obtain accelerated methods for convex unconstrained minimization in different settings. As an application of the general scheme we propose nearly optimal methods for minimizing smooth functions with Lipschitz derivatives of an arbitrary order, as well as for smooth minimax optimization problems. The proposed meta-algorithm is more general than the ones in the literature and allows to obtain better convergence rates and practical performance in several settings.

Superlinear convergence of Broyden’s method and BFGS algorithm using Kantorovich-type assumptions

Broyden’s method is a quasi-Newton method which is used to solve a system of nonlinear equations. Almost all convergence theories in the literature assume existence of a root and bounds on the nonlinear function and its derivative in some neighbourhood of the root. All these conditions cannot be checked in practice. The motivation of this work is to derive a local convergence theory where all assumptions can be verified, and the existence of a root and its superlinear rate of convergence are consequences of the theory. The BFGS algorithm is a quasi-Newton method for unconstrained minimization. Also, all known convergence theories assume existence of a solution and bounds of the function in a neighbourhood of the minimizer. The second main result of this paper is a local convergence theory where all assumptions are verifiable and existence of a minimizer and superlinear convergence of the iteration are conclusions. In addition, both theories are simple in the sense that they contain as few constants as possible.

An Active Set Smoothing Method for Solving Unconstrained Minimax Problems

In this paper, an active set smoothing function based on the plus function is constructed for the maximum function. The active set strategy used in the smoothing function reduces the number of gradients and Hessians evaluations of the component functions in the optimization. Combing the active set smoothing function, a simple adjustment rule for the smoothing parameters, and an unconstrained minimization method, an active set smoothing method is proposed for solving unconstrained minimax problems. The active set smoothing function is continuously differentiable, and its gradient is locally Lipschitz continuous and strongly semismooth. Under the boundedness assumption on the level set of the objective function, the convergence of the proposed method is established. Numerical experiments show that the proposed method is feasible and efficient, particularly for the minimax problems with very many component functions.

On inexact solution of auxiliary problems in tensor methods for convex optimization

ABSTRACT In this paper, we study the auxiliary problems that appear in p-order tensor methods for unconstrained minimization of convex functions with ν-Hölder continuous pth derivatives. This type of auxiliary problems corresponds to the minimization of a -order regularization of the pth-order Taylor approximation of the objective. For the case p = 3, we consider the use of Gradient Methods with Bregman distance. When the regularization parameter is sufficiently large, we prove that the referred methods take at most iterations to find either a suitable approximate stationary point of the tensor model or an ε-approximate stationary point of the original objective function.

Stochastic Three Points Method for Unconstrained Smooth Minimization

In this paper we consider the unconstrained minimization problem of a smooth function in $\mathbb{R}^n$ in a setting where only function evaluations are possible. We design a novel randomized deriv...

Tensor Methods for Minimizing Convex Functions with Hölder Continuous Higher-Order Derivatives

In this paper we study $p$-order methods for unconstrained minimization of convex functions that are $p$-times differentiable ($p\geq 2$) with $\nu$-H\"{o}lder continuous $p$th derivatives. We propose tensor schemes with and without acceleration. For the schemes without acceleration, we establish iteration complexity bounds of $\mathcal{O}\left(\epsilon^{-1/(p+\nu-1)}\right)$ for reducing the functional residual below a given $\epsilon\in (0,1)$. Assuming that $\nu$ is known, we obtain an improved complexity bound of $\mathcal{O}\left(\epsilon^{-1/(p+\nu)}\right)$ for the corresponding accelerated scheme. For the case in which $\nu$ is unknown, we present a universal accelerated tensor scheme with iteration complexity of $\mathcal{O}\left(\epsilon^{-p/[(p+1)(p+\nu-1)]}\right)$. A lower complexity bound of $\mathcal{O}\left(\epsilon^{-2/[3(p+\nu)-2]}\right)$ is also obtained for this problem class.

Affine-Projection-Like M-Estimate Adaptive Filter for Robust Filtering in Impulse Noise

In this brief, an affine-projection-like M-estimate (APLM) algorithm is proposed for robust adaptive filtering. To eliminate the adverse effects of impulsive noise in case of the impulse interference environment on the filter weight updates. The proposed APLM algorithm uses a robust cost function based on M-estimate and is derived by using the unconstrained minimization method. More importantly, the APLM algorithm has lower computational complexity than the M-estimate affine projection algorithm, since the direct or indirect inversion of the input signal matrix does not need to be calculated. In order to further improve the performance of the APLM algorithm, namely convergence speed and steady-state misalignment, the convex combination of the APLM (C-APLM) algorithm is presented. Simulation results verify that the proposed APLM and C-APLM algorithms are effective in system identification and echo cancellation scenarios. It also demonstrates that the C-APLM algorithm improves the filter performance in terms of the convergence speed and the normalized mean squared deviation in the presence of impulse noise.

Superlinear convergence of nonlinear conjugate gradient method and scaled memoryless BFGS method based on assumptions about the initial point

The Perry nonlinear conjugate gradient method and scaled memoryless BFGS method are two quasi-Newton methods for unconstrained minimization. All convergence theory in the literature assume existence of a minimizer and bounds on the objective function in a neighbourhood of the minimizer. These conditions cannot be checked in practice. The motivation of this work is to derive a convergence theory where all assumptions can be verified, and the existence of a minimizer and its superlinear rate of convergence are consequences of the theory. Only the basic versions of these methods without line search are considered. The theory is simple in the sense that it contains as few constants as possible.

Robust Variable Step-Size Reweighted Zero-Attracting Least Mean M-Estimate Algorithm for Sparse System Identification

The reweighted zero-attracting least mean square (RZA-LMS) algorithm has good performance in sparse system identification. However, the convergence performance of the RZA-LMS algorithm degrades in the impulsive noise environment. In this brief, a reweighted zero-attracting least mean M-estimate (RZA-LMM) algorithm is proposed to address the issue. The RZA-LMM algorithm employs the M-estimate cost function with a sparsity-aware penalty term and is derived by using the unconstrained minimization method. In order to further enhance the convergence speed and reduce the steady-state error of the RZA-LMM algorithm, a variable step-size reweighted zero-attracting least mean M-estimate (VSSRZA-LMM) algorithm is proposed. The variable step-size scheme, which adopts the M-estimate cost function with a sparsity-aware penalty term and a constrained term of the step-size, is obtained by using the constrained minimization method. Besides, the mean-square stability is analyzed to obtain the range of the step-size, which guarantees the stabilization of the proposed algorithms. The computer simulation results show that the proposed RZA-LMM algorithm achieves better convergence performance than the RZA-LMS algorithm, and the proposed VSSRZA-LMM algorithm exhibits better convergence performance in terms of convergence speed and steady-state normalized mean square deviation (NMSD) than the RZA-LMM algorithm for sparse system identification in the impulsive noise environment.

A Newton-like method with mixed factorizations and cubic regularization for unconstrained minimization

A Newton-like method for unconstrained minimization is introduced in the present work. While the computer work per iteration of the best-known implementations may need several factorizations or may use rather expensive matrix decompositions, the proposed method uses a single cheap factorization per iteration. Convergence and complexity and results, even in the case in which the subproblems’ Hessians are far from being Hessians of the objective function, are presented. Moreover, when the Hessian is Lipschitz-continuous, the proposed method enjoys $$O(\varepsilon ^{-3/2})$$ evaluation complexity for first-order optimality and $$O(\varepsilon ^{-3})$$ for second-order optimality as other recently introduced Newton method for unconstrained optimization based on cubic regularization or special trust-region procedures. Fairly successful and fully reproducible numerical experiments are presented and the developed corresponding software is freely available.

A flexible sparse set-membership NLMS algorithm for multi-path and acoustic echo channel estimations

A highly improved set-membership normalized least mean square (SM-NLMS) algorithm is carried out to further strengthen the estimation behavior of previously presented SM-NLMS algorithm for estimating sparse multi-path and acoustic echo channels. The newly developed sparse SM-NLMS algorithm utilizes an approximating l0 function (AL0) to modify the SM-NLMS’s cost function to devise a zero attractor. Then, a zero attractor is realized to provide a quick zero attraction for estimating the sparse signals. The proposed soft parameter functioned SM-NLMS (SPFSM-NLMS) algorithm is derived by means of the unconstrained stochastic gradient minimization method to form the finalized SPFSM-NLMS (FSPFSM-NLMS) algorithm. The FSPFSM-NLMS algorithm is derived mathematically, and whose estimation behavior is considered and analyzed on a multi-path sparse channel as well as an acoustic echo channel. Moreover, our presented FSPFSM-NLMS algorithm is also studied under various sparsity levels. The simulated results illustrate that the developed FSPFSM-NLMS algorithm provides a superior sparse channel estimation behavior compared to the zero-attracting SM-NLMS (ZASM-NLMS), reweighting ZASM-NLMS (RZASM-NLMS) and NLMS in terms of the steady-state performance and convergence speed.

M-estimate Affine Projection Algorithm Based On Correntropy Induced Metric

In this paper, an M-estimate affine projection algorithm based on correntropy induced metric (MAPA-CIM) is proposed for robust sparse adaptive filtering. The proposed MAPA-CIM algorithm uses an M-estimate robust cost function with correntropy induced metric, which is derived by using the unconstrained minimization method. Simulation results show that the proposed MAPA-CIM algorithm has better convergence speed and lower steady-state misalignment for sparse system identification and echo cancellation scenarios in non-Gaussian environments with colored input signal over the usual adaptive filtering algorithms.

Author’s response to comment on ‘application of noise guidance to the assessment of industrial noise with character on residential dwellings in the UK’, D. Baker 93 (2015) 88–96

A highly improved set-membership normalized least mean square (SM-NLMS) algorithm is carried out to further strengthen the estimation behavior of previously presented SM-NLMS algorithm for estimating sparse multi-path and acoustic echo channels. The newly developed sparse SM-NLMS algorithm utilizes an approximating l0 function (AL0) to modify the SM-NLMS’s cost function to devise a zero attractor. Then, a zero attractor is realized to provide a quick zero attraction for estimating the sparse signals. The proposed soft parameter functioned SM-NLMS (SPFSM-NLMS) algorithm is derived by means of the unconstrained stochastic gradient minimization method to form the finalized SPFSM-NLMS (FSPFSM-NLMS) algorithm. The FSPFSM-NLMS algorithm is derived mathematically, and whose estimation behavior is considered and analyzed on a multi-path sparse channel as well as an acoustic echo channel. Moreover, our presented FSPFSM-NLMS algorithm is also studied under various sparsity levels. The simulated results illustrate that the developed FSPFSM-NLMS algorithm provides a superior sparse channel estimation behavior compared to the zero-attracting SM-NLMS (ZASM-NLMS), reweighting ZASM-NLMS (RZASM-NLMS) and NLMS in terms of the steady-state performance and convergence speed.

Properties of the block BFGS update and its application to the limited-memory block BNS method for unconstrained minimization

A block version of the Broyden–Fletcher–Goldfarb–Shanno (BFGS) variable metric update formula and its modifications are investigated. In spite of the fact that this formula satisfies the quasi-Newton conditions with all used difference vectors and that the improvement of convergence is the best one in some sense for quadratic objective functions, for general functions, it does not guarantee that the corresponding direction vectors are descent directions. To overcome this difficulty, but at the same time utilize the advantageous properties of the block BFGS update, a block version of the limited-memory variable metric BNS method for large-scale unconstrained optimization is proposed. The global convergence of the algorithm is established for convex sufficiently smooth functions. Numerical experiments demonstrate the efficiency of the new method.

New unconstrained minimization methods for robust flash calculations at temperature, volume and moles specifications

In this paper, a new method for phase equilibrium calculations at constant temperature, volume and moles (VTN flash) is presented. The problem is formulated as an unconstrained minimization of the Helmholtz free energy with respect to mole numbers. Volume is a dependent variable and its dependence on mole numbers is taken into account by solving a nonlinear equation (the volume balance equation) for pressure at each iteration level. By analyzing the block structure of the Hessian matrix in the constrained minimization of the Helmholtz free energy with respect to mole numbers and volume (used in previous approaches), it is shown that the Hessian in the proposed method has a higher implicitness level and Newton iterations converge faster (in terms of number of iterations). The calculation framework is similar to that in flash calculations at pressure and temperature specifications (PT flash). A combined successive substitution - Newton method is used, with mole numbers or natural logarithms of equilibrium constants as independent variables. The iterations in the VTN flash are “PT-like” (for a sequence of pressures converging to the pressure at which the volume specification is met) and the number of iterations required for convergence is comparable to that of the PT flash performed at the final pressure. The proposed method is tested for several mixtures of various complexities and proved to be rapid and robust. For vapor-liquid equilibrium, the convergence is obtained starting from the ideal equilibrium constants, even very close to phase boundaries and at near-critical conditions. Existing codes for PT flash calculations can be easily modified to incorporate the new method for VTN flash calculations.