First-order Critical Points Research Articles

A Retrospective Approximation Approach for Smooth Stochastic Optimization

Stochastic Gradient (SG) is the de facto iterative technique to solve stochastic optimization (SO) problems with a smooth (nonconvex) objective f and a stochastic first-order oracle. SG’s attractiveness is due in part to its simplicity of executing a single step along the negative subsampled gradient direction to update the incumbent iterate. In this paper, we question SG’s choice of executing a single step as opposed to multiple steps between subsample updates. Our investigation leads naturally to generalizing SG into Retrospective Approximation (RA), where, during each iteration, a “deterministic solver” executes possibly multiple steps on a subsampled deterministic problem and stops when further solving is deemed unnecessary from the standpoint of statistical efficiency. RA thus formalizes what is appealing for implementation—during each iteration, “plug in” a solver—for example, L-BFGS line search or Newton-CG—as is, and solve only to the extent necessary. We develop a complete theory using relative error of the observed gradients as the principal object, demonstrating that almost sure and L1 consistency of RA are preserved under especially weak conditions when sample sizes are increased at appropriate rates. We also characterize the iteration and oracle complexity (for linear and sublinear solvers) of RA and identify a practical termination criterion leading to optimal complexity rates. To subsume nonconvex f, we present a certain “random central limit theorem” that incorporates the effect of curvature across all first-order critical points, demonstrating that the asymptotic behavior is described by a certain mixture of normals. The message from our numerical experiments is that the ability of RA to incorporate existing second-order deterministic solvers in a strategic manner might be important from the standpoint of dispensing with hyper-parameter tuning. Funding: R. Pasupathy received financial support from the Office of Naval Research [Grants N000141712295 and 13000991]. R. Bollapragada received financial support from the Lawrence Livermore National Laboratory and the National Science Foundation [Grant NSF DMS 2324643].

Efficient trust region filter modeling strategies for computationally expensive black-box optimization

The study and application of contemporary optimization techniques considerably enhance the efficiency of chemical research and manufacturing. With the dynamic progression of modern manufacturing technologies, the emergence of numerous black-box models characterized by inaccessible mathematical formulations and high evaluation costs poses new challenges to traditional optimization methods, leading to difficulty in programming and solving. Hence, based on the trust region filter (TRF) method, we define a new framework to elevate optimization efficiency in this study for chemical systems involving computationally expensive black-box functions. Sampling size per iteration is reduced, and sampling efficiency is improved by incorporating the known data beyond the trust region to assist in constructing reduced models, which is achieved by the application of the Gaussian process. Through comparison and validation of benchmark tests and case studies, this approach demonstrates that using the Gaussian process as the reduced model can lower the number of calls to black-box functions by more than half compared to common linear and quadratic models, and convergence to first-order critical points can still be guaranteed.

Iteration complexity and finite-time efficiency of adaptive sampling trust-region methods for stochastic derivative-free optimization

ASTRO-DF is a prominent trust-region method using adaptive sampling for stochastic derivative-free optimization of nonconvex problems. Its salient feature is an easy-to-understand-and-implement concept of maintaining “just enough” replications when evaluating points throughout the search to guarantee almost-sure convergence to a first-order critical point. To reduce the dependence of ASTRO-DF on the problem dimension and boost its performance in finite time, we present two key refinements, namely: (i) local models with diagonal Hessians constructed on interpolation points based on a coordinate basis; and (ii) direct search using the interpolation points whenever possible. We demonstrate that the refinements in (i) and (ii) retain the convergence guarantees while matching existing results on iteration complexity. Uniquely, our iteration complexity results match the canonical rates without placing assumptions on iterative models’ quality and their independence from function estimates. Numerical experimentation on a testbed of problems and comparison against existing popular algorithms reveals the computational advantage of ASTRO-DF due to the proposed refinements.

An Improved WENO-Z Scheme for Hyperbolic Conservation Laws with New Global Smoothness Indicator

The fifth-order WENO-Z scheme proposed by Borges et al., using a linear combination of low-order smoothness indicators, is designed to provide a low numerical dissipation to solve hyperbolic conservation laws, while the power q in the framework of WENO-Z plays a key role in its performance. In this paper, a novel global smoothness indicator with fifth-order accuracy, which is based on several lower-order smoothness indicators on two-point sub-stencils, is presented, and a new lower-dissipation WENO-Z scheme (WENO-NZ) is developed. The spectral properties of the WENO-NZ scheme are studied through the ADR method and show that this new scheme can exhibit better spectral results than WENO-Z no matter what the power value is. Accuracy tests confirm that the accuracy of WENO-Z with q = 1 would degrade to the fourth order at first-order critical points, while WENO-NZ can recover the optimal fifth-order convergence. Furthermore, numerical experiments with one- and two-dimensional benchmark problems demonstrate that the proposed WENO-NZ scheme can efficiently decrease the numerical dissipation and has a higher resolution compared to the WENO-Z scheme.

Third-order scale-independent WENO-Z scheme to achieve optimal order at critical points

Our past work has shown that when the critical points occur within grid intervals, the relations of accuracy of the smoothness indicators of weighted essentially non-oscillatory schemes (WENO) reported by Jiang and Shu differ from those that are obtained by assuming that the critical points occur on the grid nodes. The global smoothness indicator in the WENO-Z scheme might accordingly differ from the original one. We use this understanding to first discuss several issues regarding current improvements to third-order WENO-Z (e.g., WENO-NP3, -F3, -NN3, and -PZ3), i.e., numerical results with scale dependence, the validity of the analysis by assuming that the critical points occur on the nodes, and sensitivity in terms of the computational time step and initial conditions through the examination of the order of convergence. Numerical simulations and analyses were used to highlight defects in these improvements that occur either due to the scale dependence of the results, or the failure to recover the optimal order when the critical points occur on the half-nodes. Following this, a generic analysis that assumes that the first-order critical points occur within the grid intervals is provided. The theoretical results thus derived are used to propose two scale-independent third-order WENO-Z schemes that can be used to attain the optimal order at the critical points. The first scheme is obtained by extending a downstream smoothness indicator to derive a new global smoothness indicator and incorporating it into the mapping function. The second scheme is achieved by extending another smoothness indicator and using a different global indicator. The following validations are chosen and tested: the typical 1D problem of scalar advections, and 1D and 2D problems based on Euler's equations. The results verify the capability of the proposed schemes to recover the optimal order at the critical points. Moreover, the first of the above two proposed schemes outperforms the improved third-order WENO-Z scheme in terms of numerical resolution and robustness, which is usually favored by applications.

An Accelerated First-Order Method for Non-convex Optimization on Manifolds

We describe the first gradient methods on Riemannian manifolds to achieve accelerated rates in the non-convex case. Under Lipschitz assumptions on the Riemannian gradient and Hessian of the cost function, these methods find approximate first-order critical points faster than regular gradient descent. A randomized version also finds approximate second-order critical points. Both the algorithms and their analyses build extensively on existing work in the Euclidean case. The basic operation consists in running the Euclidean accelerated gradient descent method (appropriately safe-guarded against non-convexity) in the current tangent space, then moving back to the manifold and repeating. This requires lifting the cost function from the manifold to the tangent space, which can be done for example through the Riemannian exponential map. For this approach to succeed, the lifted cost function (called the pullback) must retain certain Lipschitz properties. As a contribution of independent interest, we prove precise claims to that effect, with explicit constants. Those claims are affected by the Riemannian curvature of the manifold, which in turn affects the worst-case complexity bounds for our optimization algorithms.

A new nonlocal low-rank regularization method with applications to magnetic resonance image denoising

Magnetic resonance (MR) images are frequently corrupted by Rician noise during image acquisition and transmission. And it is very challenging to restore MR data because Rician noise is signal-dependent. By exploring the nonlocal self-similarity of natural images and further using the low-rank prior of the matrices formed by nonlocal similar patches for 2D data or cubes for 3D data, we propose in this paper a new nonlocal low-rank regularization (NLRR) method including an optimization model and an efficient iterative algorithm to remove Rician noise. The proposed mathematical model consists of a data fidelity term derived from a maximum a posteriori estimation and a NLRR term using the log-det function. The resulting model in terms of approximated patch/cube matrices is non-convex and non-smooth. To solve this model, we propose an alternating reweighted minimization (ARM) algorithm using the Lipschitz-continuity of the gradient of the fidelity term and the concavity of the logarithmic function in the log-det function. The subproblems of the ARM algorithm have closed-form solutions and its limit points are first-order critical points of the problem. The ARM algorithm is further integrated with a two-stage scheme to enhance the denoising performance of the proposed NLRR method. Experimental results tested on 2D and 3D MR data, including simulated and real data, show that the NLRR method outperforms existing state-of-the-art methods for removing Rician noise.

On Developing Piecewise Rational Mapping with Fine Regulation Capability for WENO Schemes

On the idea of mapped WENO scheme, properties of mapping methods are analyzed, uncertainties in mapping development are investigated, and new piecewise rational mappings are proposed. Based on our former understandings, i.e. the mapping at endpoints {0, 1} tending to identity mapping, a so-called Cn,m condition is summarized for function development. Uncertainties, i.e., whether the pattern at endpoints of mapping would make mapped scheme behave like WENO or ENO, whether piecewise implementation of mapping would entail numerical instability, and whether WENO3 could preserve the third-order at first-order critical points by mapping, are analyzed and clarified. A new piecewise rational mapping with sufficient regulation capability is developed afterwards, where the flatness of mapping around the linear weights and the profile at endpoint tending toward identity mapping can be coordinated explicitly and simultaneously. Hence, the increase of resolution and preservation of stability can be balanced. Especially, concrete mappings are determined for {WENO3, 5, 7}. Numerical examples are tested for the new mapped WENO, which regard preservation and convergence rate of accuracy, numerical stability including that in the long-time computation, resolution and robustness. For comparison, some recent mappings such as IM by [App. Math. Comput. 232, 2014:453–468], RM by [J. Sci. Comput. 67, 2016:540–580] and AIM by [J. Comput. Phys. 381, 2019:162–188] are tested; in addition, some recent WENO-Z type schemes are chosen as well. The results manifest that new schemes can preserve optimal orders at corresponding critical points, achieve numerical stability, and indicate overall comparative advantages regarding accuracy, resolution and robustness.

Second-Order Guarantees of Distributed Gradient Algorithms

We consider distributed smooth nonconvex unconstrained optimization over net- works, modeled as a connected graph. We examine the behavior of distributed gradient-based algorithms near strict saddle points. Specifically, we establish that (i) the renowned distributed gradient descent algorithm likely converges to a neighborhood of a second-order stationary (SoS) solution; and (ii) the more recent class of distributed algorithms based on gradient tracking---implementable also over digraphs---likely converges to exact SoS solutions, thus avoiding (strict) saddle points. Furthermore, new convergence rate results for first-order critical points is established for the latter class of algorithms.

Modified Stencil Approximations for Fifth-Order Weighted Essentially Non-oscillatory Schemes

In this paper, a modified fifth-order weighted essentially non-oscillatory (WENO) finite difference scheme is presented. The quadratic polynomial approximation of numerical flux on each candidate stencil of the traditional WENO-JS scheme is modified by adding a form of cubic terms such that the resulting stencil approximation achieves fourth-order accuracy. And the corresponding smoothness indicators are calculated. The modified candidate fluxes and local smoothness indicators, when used in the WENO-JS scheme, can make the resulting new scheme (called WENO-MS) achieve fifth-order convergence in smooth regions including first-order critical points. A series of one- and two-dimensional numerical examples are presented to demonstrate the performance of the new scheme. The numerical results show that the proposed WENO-MS scheme provides a comparable or higher resolution of fine structures compared with the WENO-M, WENO-Z and P-WENO schemes, while it increases only 7% of CPU time compared with the traditional WENO-JS scheme.

Convergence and evaluation-complexity analysis of a regularized tensor-Newton method for solving nonlinear least-squares problems

Given a twice-continuously differentiable vector-valued function r(x), a local minimizer of $$\Vert r(x)\Vert _2$$ is sought. We propose and analyse tensor-Newton methods, in which r(x) is replaced locally by its second-order Taylor approximation. Convergence is controlled by regularization of various orders. We establish global convergence to a first-order critical point of $$\Vert r(x)\Vert _2$$ , and provide function evaluation bounds that agree with the best-known bounds for methods using second derivatives. Numerical experiments comparing tensor-Newton methods with regularized Gauss–Newton and Newton methods demonstrate the practical performance of the newly proposed method.

On the Convergence Analysis of Cubic Regularized Symmetric Rank-1 Quasi-Newton Method and the Incremental Version in the Application of Large-Scale Problems

This paper provides a detailed study on the convergence properties of the cubic regularized symmetric rank-1 (SR1) method (CuREG-SR1) proposed in [2]. The main advantage of incorporating cubic regularization technique to SR1 is to alleviate the problem of indefinite resulting matrix in SR1. However, its convergence under the line search framework is less studied. Here, we first show that CuREG-SR1 converges to a first-order critical point. Moreover, we give a novel result that provided the uniformly independent assumption, the difference between approximated Hessian matrix generated by CuREG-SR1 and the true Hessian is bounded. In addition, we show that for a problem with dimension d, CuREG-SR1 generates q-d superlinear steps every q iterations. We also propose a novel incremental CuREG-SR1 (ICuREG-SR1) algorithm to adapt SR1 and CuREG-SR1 efficiently for solving large scale problems. The basic idea is to incorporate incremental optimization scheme, which updates progressively information for objective function involving a sum of individual functions, which are frequently encountered in large-scale machine learning. Numerical experiments on several machine learning problems show that the proposed algorithm offers superior performance in terms of the gradient magnitude than other conventional algorithms tested.

On the Global Convergence of a Projective Trust Region Algorithm for Nonlinear Equality Constrained Optimization

A trust-region sequential quadratic programming (SQP) method is developed and analyzed for the solution of smooth equality constrained optimization problems. The trust-region SQP algorithm is based on filter line search technique and a composite-step approach, which decomposes the overall step as sum of a vertical step and a horizontal step. The algorithm includes critical modifications of horizontal step computation. One orthogonal projective matrix of the Jacobian of constraint functions is employed in trust-region subproblems. The orthogonal projection gives the null space of the transposition of the Jacobian of the constraint function. Theoretical analysis shows that the new algorithm retains the global convergence to the first-order critical points under rather general conditions. The preliminary numerical results are reported.

Evolution of structure and superconductivity in Ba(Ni1−xCox)2As2

The effects of Co-substitution on Ba(Ni$_{1-x}$Co$_x$)$_2$As$_2$ ($0\leq x\leq 0.251$) single crystals grown out of Pb flux are investigated via transport, magnetic, and thermodynamic measurements. BaNi$_2$As$_2$ exhibits a first order tetragonal to triclinic structural phase transition at $T_s=137 K$ upon cooling, and enters a superconducting phase below $T_c=0.7 K$. The structural phase transition is sensitive to cobalt content and is suppressed completely by $x\geq0.133$. The superconducting critical temperature, $T_c$, increases continuously with $x$, reaching a maximum of $T_c=2.3 K$ at the structural critical point $x=0.083$ and then decreases monotonically until superconductivity is no longer observable well into the tetragonal phase. In contrast to similar BaNi$_2$As$_2$ substitutional studies, which show an abrupt change in $T_c$ at the triclinic-tetragonal boundary that extends far into the tetragonal phase, Ba(Ni$_{1-x}$Co$_x$)$_2$As$_2$ exhibits a dome-like phase diagram centered around the first-order critical point. Together with an anomalously large heat capacity jump $\Delta C_e/\gamma T\sim 2.2$ at optimal doping, the smooth evolution of $T_c$ in the Ba(Ni$_{1-x}$Co$_x$)$_2$As$_2$ system suggests a mechanism for pairing enhancement other than phonon softening.

Many-body depletion forces of colloids in a polydisperse polymer dispersant in the long-chain limit.

We study a system of spherical non-adsorbing particles immersed in a polydisperse polymer fluid. We derive an analytic expression for the many-body depletion interactions between the colloidal particles in the limit of very long chains. We argue that this expression is essentially exact for long chains and justify this using explicit simulations. In this way we are able to elucidate the profound effect of many-body interactions on the particle thermodynamics. We show that using truncated 2-body depletion interactions leads to strong particle segregation, while the complete many-body description predicts that the total depletion force becomes weak so that the system approximates one which interacts via a so-called Kac potential. This implies that the depletion interactions can be described using mean-field theory. We show that many-body effects cause a significant contraction of the 2-phase region of the particle dispersion. We also investigate the system approaching the (tricritical) θ point, which terminates the line of first-order critical points of the polymer dispersion in a poor solvent and show that many-body effects suppress particle phase transitions.

ASTRO-DF: A Class of Adaptive Sampling Trust-Region Algorithms for Derivative-Free Stochastic Optimization

We consider unconstrained optimization problems where only “stochastic” estimates of the objective function are observable as replicates from a Monte Carlo oracle. The Monte Carlo oracle is assumed to provide no direct observations of the function gradient. We present ASTRO-DF---a class of derivative-free trust-region algorithms, where a stochastic local model is constructed, optimized, and updated iteratively. Function estimation and model construction within ASTRO-DF is adaptive in the sense that the extent of Monte Carlo sampling is determined by continuously monitoring and balancing measures of sampling error (or variance) and structural error (or model bias) within ASTRO-DF. Such balancing of errors is designed to ensure that Monte Carlo effort within ASTRO-DF is sensitive to algorithm trajectory: sampling is higher whenever an iterate is inferred to be close to a critical point and lower when far away. We demonstrate the almost sure convergence of ASTRO-DF's iterates to first-order critical points when using stochastic polynomial interpolation models. The question of using more complicated models, e.g., regression or stochastic kriging, in combination with adaptive sampling is worth further investigation and will benefit from the methods of proof presented here. We speculate that ASTRO-DF's iterates achieve the canonical Monte Carlo convergence rate, although a proof remains elusive.

Worst-case evaluation complexity of regularization methods for smooth unconstrained optimization using Hölder continuous gradients

The worst-case behaviour of a general class of regularization algorithms is considered in the case where only objective function values and associated gradient vectors are evaluated. Upper bounds are derived on the number of such evaluations that are needed for the algorithm to produce an approximate first-order critical point whose accuracy is within a user-defined threshold. The analysis covers the entire range of meaningful powers in the regularization term as well as in the Hölder exponent for the gradient. The resulting complexity bounds vary according to the regularization power and the assumed Hölder exponent, recovering known results when available.

Variable sample size method for equality constrained optimization problems

An equality constrained optimization problem with a deterministic objective function and constraints in the form of mathematical expectation is considered. The constraints are transformed into the Sample Average Approximation form resulting in deterministic problem. A method which combines a variable sample size procedure with line search is applied to a penalty reformulation. The method generates a sequence that converges towards first-order critical points. The final stage of the optimization procedure employs the full sample and the SAA problem is eventually solved with significantly smaller cost. Preliminary numerical results show that the proposed method can produce significant savings compared to SAA method and some heuristic sample update counterparts while generating a solution of the same quality.

A New Restarting Adaptive Trust-Region Method for Unconstrained Optimization

In this paper, we present a new adaptive trust-region method for solving nonlinear unconstrained optimization problems. More precisely, a trust-region radius based on a nonmonotone technique uses an approximation of Hessian which is adaptively chosen. We produce a suitable trust-region radius; preserve the global convergence under classical assumptions to the first-order critical points; improve the practical performance of the new algorithm compared to other exiting variants. Moreover, the quadratic convergence rate is established under suitable conditions. Computational results on the CUTEst test collection of unconstrained problems are presented to show the effectiveness of the proposed algorithm compared with some exiting methods.

Worst-case evaluation complexity for unconstrained nonlinear optimization using high-order regularized models

The worst-case evaluation complexity for smooth (possibly nonconvex) unconstrained optimization is considered. It is shown that, if one is willing to use derivatives of the objective function up to order p (for $$p\ge 1$$pź1) and to assume Lipschitz continuity of the p-th derivative, then an $$\epsilon $$∈-approximate first-order critical point can be computed in at most $$O(\epsilon ^{-(p+1)/p})$$O(∈-(p+1)/p) evaluations of the problem's objective function and its derivatives. This generalizes and subsumes results known for $$p=1$$p=1 and $$p=2$$p=2.