Approximate Critical Point Research Articles

Optimization Algorithms for Multi-species Spherical Spin Glasses

This paper develops approximate message passing algorithms to optimize multi-species spherical spin glasses. We first show how to efficiently achieve the algorithmic threshold energy identified in our companion work (Huang and Sellke in arXiv preprint, 2023. arXiv:2303.12172), thus confirming that the Lipschitz hardness result proved therein is tight. Next we give two generalized algorithms which produce multiple outputs and show all of them are approximate critical points. Namely, in an r-species model we construct 2r\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$2^r$$\\end{document} approximate critical points when the external field is stronger than a “topological trivialization" phase boundary, and exponentially many such points in the complementary regime. We also compute the local behavior of the Hamiltonian around each. These extensions are relevant for another companion work (Huang and Sellke in arXiv preprint, 2023. arXiv:2308.09677) on topological trivialization of the landscape.

A redistributed proximal bundle method for nonsmooth nonconvex functions with inexact information

In this paper, we propose a redistributed proximal bundle method for a class of nonconvex nonsmooth optimization problems with inexact information, i.e., we consider the problem of computing the approximate critical points when only the inexact information about the function values and subgradients are available and show that reasonable convergence properties are obtained. We assume that the errors in the computation of functions and subgradients are only bounded and in principle do not have to vanish within the limits. For the nonconvex functions, we design the convexification technique, which ensures that the linearization error of its augmentation function is nonnegative. Meanwhile, for the inexact information, we utilize noise management strategies and update approximate parameters to reduce the impact of inexact information. Based on this method, we can obtain the approximate solution.

First-order inertial algorithms involving dry friction damping

In a Hilbert space $$ {\mathcal H}$$ , we introduce a new class of first-order algorithms which naturally occur as discrete temporal versions of an inertial differential inclusion jointly involving viscous friction and dry friction. The function $$f:{\mathcal H}\rightarrow {\mathbb {R}}$$ to be minimized is supposed to be differentiable (not necessarily convex), and enters the algorithm via its gradient. The dry friction damping function $$\phi :{\mathcal H}\rightarrow {\mathbb {R}}_+$$ is convex with a sharp minimum at the origin, (typically $$\phi (x) = r \Vert x\Vert $$ with $$r >0$$ ). It enters the algorithm via its proximal mapping, which acts as a soft threshold operator on the velocities. As a result, we obtain a new class of splitting algorithms involving separately the proximal and gradient steps. The sequence of iterates has a finite length, and therefore strongly converges towards an approximate critical point $$x_{\infty }$$ of f (typically $$\Vert \nabla f(x_{\infty })\Vert \le r$$ ). Under a geometric property satisfied by the limit point $$x_{\infty }$$ , we obtain geometric and finite rates of convergence. The convergence results tolerate the presence of errors, under the sole assumption of their asymptotic convergence towards zero. By replacing the function f by its Moreau envelope, we extend the results to the case of nonsmooth convex functions. In this case, the algorithm involves the proximal operators of f and $$\phi $$ separately. Several variants of this algorithm are considered, including the case of the Nesterov accelerated gradient method. We then consider the extension in the case of additive composite optimization, thus leading to new splitting methods. Numerical experiments are given for Lasso-type problems. The performance profiles, as a comparison tool, demonstrate the efficiency of the Nesterov accelerated method with asymptotic vanishing damping combined with dry friction.

Sequential Difference-of-Convex Programming

Optimization methods for difference-of-convex programs iteratively solve convex subproblems to define iterates. Although convex, depending on the problem’s structure, these subproblems are very often challenging and require specialized solvers. This work investigates a new methodology that defines iterates as approximate critical points of significantly easier difference-of-convex subproblems approximating the original one. Since there is considerable freedom to choose such more accessible subproblems, several algorithms can be designed from the given approach. In some cases, the resulting algorithm boils down to a straightforward process with iterates given in an analytic form. In other situations, decomposable subproblems can be chosen, opening the way for parallel computing even when the original program is not decomposable. Depending on the problem’s assumptions, a possible variant of the given approach is the Josephy–Newton method applied to the system of (necessary) optimality conditions of the original difference-of-convex program. In such a setting, local convergence with superlinear and even quadratic rates can be achieved under certain conditions.

NOISY MATRIX COMPLETION: UNDERSTANDING STATISTICAL GUARANTEES FOR CONVEX RELAXATION VIA NONCONVEX OPTIMIZATION.

This paper studies noisy low-rank matrix completion: given partial and noisy entries of a large low-rank matrix, the goal is to estimate the underlying matrix faithfully and efficiently. Arguably one of the most popular paradigms to tackle this problem is convex relaxation, which achieves remarkable efficacy in practice. However, the theoretical support of this approach is still far from optimal in the noisy setting, falling short of explaining its empirical success. We make progress towards demystifying the practical efficacy of convex relaxation vis-à-vis random noise. When the rank and the condition number of the unknown matrix are bounded by a constant, we demonstrate that the convex programming approach achieves near-optimal estimation errors - in terms of the Euclidean loss, the entrywise loss, and the spectral norm loss - for a wide range of noise levels. All of this is enabled by bridging convex relaxation with the nonconvex Burer-Monteiro approach, a seemingly distinct algorithmic paradigm that is provably robust against noise. More specifically, we show that an approximate critical point of the nonconvex formulation serves as an extremely tight approximation of the convex solution, thus allowing us to transfer the desired statistical guarantees of the nonconvex approach to its convex counterpart.

Robust Euclidean embedding via EDM optimization

This paper aims to propose an efficient numerical method for the most challenging problem known as the robust Euclidean embedding (REE) in the family of multi-dimensional scaling (MDS). The problem is notoriously known to be nonsmooth, nonconvex and its objective is non-Lipschitzian. We first explain that the semidefinite programming (SDP) relaxations and Euclidean distance matrix (EDM) approach, popular for other types of problems in the MDS family, failed to provide a viable method for this problem. We then propose a penalized REE (PREE), which can be economically majorized. We show that the majorized problem is convex provided that the penalty parameter is above certain threshold. Moreover, it has a closed-form solution, resulting in an efficient algorithm dubbed as PREEEDM (for Penalized REE via EDM optimization). We prove among others that PREEEDM converges to a stationary point of PREE, which is also an approximate critical point of REE. Finally, the efficiency of PREEEDM is compared with several state-of-the-art methods including SDP and EDM solvers on a large number of test problems from sensor network localization and molecular conformation.

The Discrete-Time Geometric Maximum Principle

We establish a variety of results extending the well-known Pontryagin maximum principle of optimal control to discrete-time optimal control problems posed on smooth manifolds. These results are organized around a new theorem on critical and approximate critical points for discrete-time geometric control systems. We show that this theorem can be used to derive Lie group variational integrators in Hamiltonian form; to establish a maximum principle for control problems in the absence of state constraints; and to provide sufficient conditions for exact penalization techniques in the presence of state or mixed constraints. Exact penalization techniques are used to study sensitivity of the optimal value function to constraint perturbations and to prove necessary conditions for optimality, including in the form of a maximum principle, for discrete-time geometric control problems with state or mixed constraints.

Second-Order Optimality and Beyond: Characterization and Evaluation Complexity in Convexly Constrained Nonlinear Optimization

High-order optimality conditions for convexly constrained nonlinear optimization problems are analysed. A corresponding (expensive) measure of criticality for arbitrary order is proposed and extended to define high-order epsilon -approximate critical points. This new measure is then used within a conceptual trust-region algorithm to show that if derivatives of the objective function up to order q ge 1 can be evaluated and are Lipschitz continuous, then this algorithm applied to the convexly constrained problem needs at most O(epsilon ^{-(q+1)}) evaluations of f and its derivatives to compute an epsilon -approximate qth-order critical point. This provides the first evaluation complexity result for critical points of arbitrary order in nonlinear optimization. An example is discussed, showing that the obtained evaluation complexity bounds are essentially sharp.

A proximal bundle method for nonsmooth nonconvex functions with inexact information

For a class of nonconvex nonsmooth functions, we consider the problem of computing an approximate critical point, in the case when only inexact information about the function and subgradient values is available. We assume that the errors in function and subgradient evaluations are merely bounded, and in principle need not vanish in the limit. We examine the redistributed proximal bundle approach in this setting, and show that reasonable convergence properties are obtained. We further consider a battery of difficult nonsmooth nonconvex problems, made even more difficult by introducing inexactness in the available information. We verify that very satisfactory outcomes are obtained in our computational implementation of the inexact algorithm.

Identifying Activity

Identification of active constraints in constrained optimization is of interest from both practical and theoretical viewpoints, as it holds the promise of reducing an inequality-constrained problem to an equality-constrained problem, in a neighborhood of a solution. We study this issue in the more general setting of composite nonsmooth minimization, in which the objective is a composition of a smooth vector function with a lower semicontinuous function , typically nonsmooth but structured. In this setting, the graph of the generalized gradient can often be decomposed into a (nondisjoint) union of simpler subsets. “Identification” amounts to deciding which subsets of the graph are “active” in the criticality conditions at a given solution. We give conditions under which any convergent sequence of approximate critical points finitely identifies the activity. Prominent among these properties is a condition akin to the Mangasarian–Fromovitz constraint qualification, which ensures boundedness of the set of multiplier vectors that satisfy the optimality conditions at the solution.

A novel trace test for the mean parameters in a multivariate growth curve model

A trace test for the mean parameters of the growth curve model is proposed. It is constructed using the restricted maximum likelihood followed by an estimated likelihood ratio approach. The statistic reduces to the Lawley–Hotelling trace test for the Multivariate Analysis of Variance (MANOVA) models. Our test statistic is, therefore, a natural extension of the classical trace test to GMANOVA models. We show that the distribution of the test under the null hypothesis does not depend on the unknown covariance matrix Σ . We also show that the distributions under the null and alternative hypotheses can be represented as sums of weighted central and non-central chi-square random variables, respectively. Under the null hypothesis, the Satterthwaite approximation is used to get an approximate critical point. A novel Satterthwaite type approximation is proposed to obtain an approximate power. A simulation study is performed to evaluate the performance of our proposed test and numerical examples are provided as illustrations.

C-means clustering with the l/sub l/ and l/sub infinity / norms

An extension of the hard and fuzzy c-means (HCM/FCM) clustering algorithms is described. Specifically, these models are extended to admit the case where the (dis)similarity measure on pairs of numerical vectors includes two members of the Minkowski or p-norm family, viz., the p=1 and p= infinity norms. In the absence of theoretically necessary conditions to guide a numerical solution of the nonlinear constrained optimization problem associated with this case, it is shown that a certain basis exchange algorithm can be used to find approximate critical points of the new objective functions. This method broadens the applications horizon of the FCM family by enabling users to match discontinuous multidimensional numerical data structures with similarity measures that have nonhyperelliptical topologies.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

The chiral transition in an SU(2) gauge-Higgs-fermion theory with Yukawa couplings

We study the (zero-temperature) chiral transition in an SU(2) lattice gauge theory with I = 1 2 Higgs fields, and fermion fields with I = 1 2 and I = 0 , including Yukawa couplings. Using analytic methods in the strong gauge coupling limit, we demonstrate that (i) if at least one of the Yukawa couplings is sufficiently weak, this theory undergoes a spontaneous chiral symmetry breaking phase transition; (ii) this transition is removed when the Yukawa couplings are sufficiently strong. The transition is determined to be of second order, and approximate critical point calculations are given. Some implications for the continuum limit of the SU(2) model, and for electroweak interactions, are discussed.

K-Sample Anderson–Darling Tests

Abstract Two k-sample versions of an Anderson–Darling rank statistic are proposed for testing the homogeneity of samples. Their asymptotic null distributions are derived for the continuous as well as the discrete case. In the continuous case the asymptotic distributions coincide with the (k – 1)-fold convolution of the asymptotic distribution for the Anderson–Darling one-sample statistic. The quality of this large sample approximation is investigated for small samples through Monte Carlo simulation. This is done for both versions of the statistic under various degrees of data rounding and sample size imbalances. Tables for carrying out these tests are provided, and their usage in combining independent one- or k-sample Anderson–Darling tests is pointed out. The test statistics are essentially based on a doubly weighted sum of integrated squared differences between the empirical distribution functions of the individual samples and that of the pooled sample. One weighting adjusts for the possibly different sample sizes, and the other is inside the integration placing more weight on tail differences of the compared distributions. The two versions differ mainly in the definition of the empirical distribution function. These tests are consistent against all alternatives. The use of these tests is two-fold: (a) in a one-way analysis of variance to establish differences in the sampled populations without making any restrictive parametric assumptions or (b) to justify the pooling of separate samples for increased sample size and power in further analyses. Exact finite sample mean and variance formulas for one of the two statistics are derived in the continuous case. It appears that the asymptotic standardized percentiles serve well as approximate critical points of the appropriately standardized statistics for individual sample sizes as low as 5. The application of the tests is illustrated with an example. Because of the convolution nature of the asymptotic distribution, a further use of these critical points is possible in combining independent Anderson–Darling tests by simply adding their test statistics.

Simultaneous Confidence Bounds in Multiple Regression Using Predictor Variable Constraints

Abstract This article presents a method for approximating the coverage probability of simultaneous confidence bounds for multiple regression functions when the vector of predictor variables is constrained to lie in a polyhedral convex set. The method is useful because it allows one to construct simultaneous confidence intervals with prescribed coverage probability for the regression function evaluated at various settings of the predictor variables, which are narrower than bounds obtained without using the predictor variable constraints. For a family of two-sided simultaneous confidence bounds that includes Scheffé-type and constant-width bounds, the probability of coverage is related to the distribution of the maximum Euclidean norm of the projection onto a polyhedral cone for a pair of random vectors with known joint distribution. An analogous relation holds for one-sided bounds. If an algorithm for computing the projection onto the cone is available, then these results enable one to use the Monte Carlo method to approximate the coverage probabilities of bounds. Particular attention is given to the case in which lower and upper bounds can be specified for each of the predictor variables so that the constraint region for bounding the regression function is rectangular. An efficient algorithm for calculating projections onto the appropriate cone for rectangular constraint regions, which facilitates coverage probability approximation of Schefffé-type bounds, is presented. This algorithm is used to calculate approximate critical points of Scheffé-type bounds for some specific designs and constraint regions and we show that substantial improvements for one-sided bounds and modest improvements for two-sided bounds over the conservative bounds of Casella and Strawderman (1980) may be obtained.

Testing Subhypotheses in the Multiplicative Interaction Model

The problem of analyzing a two-way cross-classified treatment structure with only one observation per treatment combination is considered. A test procedure is given that will enable the data analyst to determine subareas of the data in which the data are additive. The procedure is developed by assuming that a multiplicative interaction model adequately fits the data. Such a model is given by y ij , = μ + τ i + β j + λα i γ j + ∊ ij ; where i = 1, 2, …, t and j = 1, 2, …, b. It is assumed that the ∊ ij , are distributed independently and normally with mean zero and variance σ2. The other parameters are assumed to be unknown constants. In general, the problem may be stated as one of testing H 0,: H α = 0 versus H a ,: H α ≠ 0, where α = (α1, …, α t )′ and H is a q × t contrast matrix. A likelihood ratio statistic for this testing problem is derived and approximate critical points are given for the cases q = 1 and q = 2. The procedures are illustrated with an example.

Tests for the homogeneity of a set of variances against ordered alternatives

SuIMMAPRY Several statistics are investigated for testing the homogeneity of a set of variances of normal populations against ordered alternatives. The null distributions of these statistics are studied. Comparisons of powers of the tests are also made by Monte Carlo experiments. The test based on the modified Bartlett's statistic M* is found to perform best. For testing the homogeneity of variances of several normal distributions test statistics such as M by Bartlett (1937), Fmax by Hartley (1950) and G by Cochran (1941) have been proposed. These test statistics are appropriate when the alternative hypothesis is merely that the variances are not all equal. In this paper we consider the problem of testing against ordered alternatives. In ? 2 the modified M, the modified Fmax and the modified G statistics based on estimates of variances under order restrictions are introduced. We derive their null distributions. The approximate critical points of the modified M are computed for 3 to 12 populations by using the method due to Hartley (1939) for approximating the null distribution of M statistic. The critical points of the modified Fmax are computed for 3 to 6 populations. In ? 3 some comparisons of the powers of tests are made by computer simulation, including the conventional test statistics mentioned above, one proposed by Vincent (1961) and a modification of it.