Non-convex Optimization Algorithm Research Articles

A New Hybrid Descent Algorithm for Large-Scale Nonconvex Optimization and Application to Some Image Restoration Problems

Conjugate gradient methods are widely used and attractive for large-scale unconstrained smooth optimization problems, with simple computation, low memory requirements, and interesting theoretical information on the features of curvature. Based on the strongly convergent property of the Dai–Yuan method and attractive numerical performance of the Hestenes–Stiefel method, a new hybrid descent conjugate gradient method is proposed in this paper. The proposed method satisfies the sufficient descent property independent of the accuracy of the line search strategies. Under the standard conditions, the trust region property and the global convergence are established, respectively. Numerical results of 61 problems with 9 large-scale dimensions and 46 ill-conditioned matrix problems reveal that the proposed method is more effective, robust, and reliable than the other methods. Additionally, the hybrid method also demonstrates reliable results for some image restoration problems.

Global convergence of dislocation hyperbolic augmented Lagrangian algorithm for nonconvex optimization

In this work, we present a new augmented Lagrangian type algorithm based on the dislocation hyperbolic augmented Lagrangian function (DHALF). This algorithm is called dislocation hyperbolic augmented Lagrangian algorithm (DHALA). We ensure that DHALA converges to a global solution for the inequality constrained nonconvex optimization problem. We present DHALA considering the safeguards technique and ensure the convergence of this algorithm. Finally, we present a computational experiment to show the performance of our proposed algorithm.

A Hybrid and Inexact Algorithm for Nonconvex and Nonsmooth Optimization

A Hybrid and Inexact Algorithm for Nonconvex and Nonsmooth Optimization

A Robust Coevolutionary Neural-Based Optimization Algorithm for Constrained Nonconvex Optimization.

For nonconvex optimization problems, a routine is to assume that there is no perturbation when executing the solution task. Nevertheless, dealing with the perturbation in advance may increase the burden on the system and take up extra time. To remedy this weakness, we propose a robust coevolutionary neural-based optimization algorithm with inherent robustness based on the hybridization between the particle swarm optimization and a class of robust neural dynamics (RND). In this framework, every neural agent guided by the RND supersedes the place of the particle, mutually searches for the optimal solution, and stabilizes itself from different perturbations. The theoretical analysis ensures that the proposed algorithm is globally convergent with probability one. Besides, the effectiveness and robustness of the proposed approach are illustrated by illustrative examples compared with the existing methods. We further apply this proposed algorithm to the source localization and manipulability optimization of the redundant manipulator, simultaneously disposing of perturbation from the internal and exogenous system with satisfactory performance.

Hessian barrier algorithms for non-convex conic optimization

AbstractA key problem in mathematical imaging, signal processing and computational statistics is the minimization of non-convex objective functions that may be non-differentiable at the relative boundary of the feasible set. This paper proposes a new family of first- and second-order interior-point methods for non-convex optimization problems with linear and conic constraints, combining logarithmically homogeneous barriers with quadratic and cubic regularization respectively. Our approach is based on a potential-reduction mechanism and, under the Lipschitz continuity of the corresponding derivative with respect to the local barrier-induced norm, attains a suitably defined class of approximate first- or second-order KKT points with worst-case iteration complexity $$O(\varepsilon ^{-2})$$ O ( ε - 2 ) (first-order) and $$O(\varepsilon ^{-3/2})$$ O ( ε - 3 / 2 ) (second-order), respectively. Based on these findings, we develop new path-following schemes attaining the same complexity, modulo adjusting constants. These complexity bounds are known to be optimal in the unconstrained case, and our work shows that they are upper bounds in the case with complicated constraints as well. To the best of our knowledge, this work is the first which achieves these worst-case complexity bounds under such weak conditions for general conic constrained non-convex optimization problems.

Online Distributed Stochastic Gradient Algorithm for Nonconvex Optimization With Compressed Communication

Online Distributed Stochastic Gradient Algorithm for Nonconvex Optimization With Compressed Communication

NGDE: A Niching-Based Gradient-Directed Evolution Algorithm for Nonconvex Optimization.

Nonconvex optimization issues are prevalent in machine learning and data science. While gradient-based optimization algorithms can rapidly converge and are dimension-independent, they may, unfortunately, fall into local optimal solutions or saddle points. In contrast, evolutionary algorithms (EAs) gradually adapt the population of solutions to explore global optimal solutions. However, this approach requires substantial computational resources to perform numerous fitness function evaluations, which poses challenges for high-dimensional optimization in particular. This study introduces a novel nonconvex optimization algorithm, the niching-based gradient-directed evolution (NGDE) algorithm, designed specifically for high-dimensional nonconvex optimization. The NGDE algorithm generates potential solutions and divides them into multiple niches to explore distinct areas within the feasible region. Subsequently, each individual creates candidate offspring using the gradient-directed mutation operator we designed. The convergence properties of the NGDE algorithm are investigated in two scenarios: accessing the full gradient and approximating the gradient with mini-batch samples. The experimental studies demonstrate the superior performance of the NGDE algorithm in minimizing multimodal optimization functions. Additionally, when applied to train the neural networks of LeNet-5, NGDE shows significantly improved classification accuracy, especially in smaller training sizes.

Dislocation hyperbolic augmented Lagrangian algorithm for nonconvex optimization

The dislocation hyperbolic augmented Lagrangian algorithm (DHALA) solves the nonconvex programming problem considering an update rule for its penalty parameter and considering a condition to ensure the complementarity condition. in this work, we ensure that the sequence generated by DHALA converges to a Karush-Kuhn-Tucker (KKT) point, and we present computational experiments to demonstrate the performance of our proposed algorithm.

Metalearning-Based Alternating Minimization Algorithm for Nonconvex Optimization.

In this article, we propose a novel solution for nonconvex problems of multiple variables, especially for those typically solved by an alternating minimization (AM) strategy that splits the original optimization problem into a set of subproblems corresponding to each variable and then iteratively optimizes each subproblem using a fixed updating rule. However, due to the intrinsic nonconvexity of the original optimization problem, the optimization can be trapped into a spurious local minimum even when each subproblem can be optimally solved at each iteration. Meanwhile, learning-based approaches, such as deep unfolding algorithms, have gained popularity for nonconvex optimization; however, they are highly limited by the availability of labeled data and insufficient explainability. To tackle these issues, we propose a meta-learning based alternating minimization (MLAM) method that aims to minimize a part of the global losses over iterations instead of carrying minimization on each subproblem, and it tends to learn an adaptive strategy to replace the handcrafted counterpart resulting in advance on superior performance. The proposed MLAM maintains the original algorithmic principle, providing certain interpretability. We evaluate the proposed method on two representative problems, namely, bilinear inverse problem: matrix completion and nonlinear problem: Gaussian mixture models. The experimental results validate the proposed approach outperforms AM-based methods.

On the Separation of Correlation-Assisted Sum Capacities of Multiple Access Channels

The capacity of a channel characterizes the maximum rate at which information can be transmitted through the channel asymptotically faithfully. For a channel with multiple senders and a single receiver, computing its sum capacity is possible in theory, but challenging in practice because of the nonconvex optimization involved. To address this challenge, we investigate three topics in our study. In the first part, we study the sum capacity of a family of multiple access channels (MACs) obtained from nonlocal games. For any MAC in this family, we obtain an upper bound on the sum rate that depends only on the properties of the game when allowing assistance from an arbitrary set of correlations between the senders. This approach can be used to prove separations between sum capacities when the senders are allowed to share different sets of correlations, such as classical, quantum or no-signalling correlations. We also construct a specific nonlocal game to show that the approach of bounding the sum capacity by relaxing the nonconvex optimization can give arbitrarily loose bounds. Owing to this result, in the second part, we study algorithms for non-convex optimization of a class of functions we call Lipschitz-like functions. This class includes entropic quantities, and hence these results may be of independent interest in information theory. Subsequently, in the third part, we show that one can use these techniques to compute the sum capacity of an arbitrary two-sender MACs to a fixed additive precision in quasi-polynomial time. We showcase our method by efficiently computing the sum capacity of a family of two-sender MACs for which one of the input alphabets has size two. Furthermore, we demonstrate with an example that our algorithm may compute the sum capacity to a higher precision than using the convex relaxation.

Federated Optimization Under Intermittent Client Availability

Federated learning is a new distributed machine learning framework, where numerous heterogeneous clients collaboratively train a model without sharing training data. In this work, we consider a practical and ubiquitous issue when deploying federated learning in mobile environments: intermittent client availability, where the set of eligible clients may change during the training process. Such intermittent client availability would seriously deteriorate the performance of the classical federated averaging algorithm (FedAvg). Thus, we propose a simple distributed nonconvex optimization algorithm, called federated latest averaging (FedLaAvg), which leverages the latest gradients of all clients, even when the clients are not available, to jointly update the global model in each iteration. Our theoretical analysis shows that FedLaAvg achieves guaranteed convergence and a sublinear speedup with respect to the total number of clients. We implement FedLaAvg along with several baselines and evaluate them over the benchmarking MNIST and Sentiment140 data sets. The evaluation results demonstrate that FedLaAvg achieves more stable training than FedAvg in both convex and nonconvex settings and reaches a sublinear speedup. Source code and online supplement are available at the IJOC GitHub site ( http://dx.doi.org/10.1287/ijoc.2022.0057.cd , https://github.com/INFORMSJoC/2022.0057 ). History: Accepted by Ram Ramesh, Area Editor for Data Science & Machine Leaning. Funding: This work was supported by the National Key R&D Program of China [Grant 2022ZD0119100], the National Natural Science Foundation of China (NSFC) [Grants 61972252, 61972254, 62072303, 62025204, 62132018, 62202296, and 62202297], the Alibaba Innovation Research (AIR) Program, and the Tencent Rhino Bird Key Research Project. Supplemental Material: The software that supports the findings of this study is available within the paper and its Supplemental Information ( https://pubsonline.informs.org/doi/suppl/10.1287/ijoc.2022.0057 ) as well as from the IJOC GitHub software repository ( https://github.com/INFORMSJoC/2022.0057 ). The complete IJOC Software and Data Repository is available at https://informsjoc.github.io/ .

Adaptive thresholding for iterative matrix completion with heterogeneous missing probability: H-AdaptiveImpute

Several low-rank matrix completion algorithms have been developed, which often employ thresholded singular value decomposition (SVD) iteratively. Their performance depends on the choice of thresholding tuning parameters and initial input matrix. In particular, the initialization of the non-convex optimization algorithms plays a pivotal role. However, they often use a zero matrix or a consistent low-rank matrix estimator under the simple homogeneous observation probability assumption as the initial input matrix. This paper proposes a non-convex low-rank matrix completion algorithm with an intelligent initial matrix under the heterogeneous observation probability structure. Specifically, the proposed algorithm iterates the thresholded SVD starting from a tailored initialization based on the estimator of the singular values and vectors of the underlying matrix. We show that the proposed initial estimator is consistent, and the adaptive thresholding tuning parameters are estimated using theoretically justified and data-driven procedures. The results of the numerical experiments show that the proposed algorithm performs well compared to the existing method.

A customized inertial proximal alternating minimization for SVD-free robust principal component analysis

Robust principal component analysis (RPCA) is devoted to tackling grossly corrupted datasets with noise. However, the performance of RPCA is usually circumscribed by the lack of efficiency of singular value decomposition (SVD), which rules out its potential applications to many large-scale real-world problems. In this paper, we develop a nonconvex optimization algorithm customized to SVD-free RPCA models. The proposed algorithm, which is built upon proximal alternating linearized minimization Bolte et al. [Proximal alternating linearized minimization for nonconvex and nonsmooth problems. Math Program. 2014;146(1–2):459–494], can reduce computational efforts by partially linearizing data fidelity and increase efficiency by leveraging inertial techniques. Under the Kurdyka-Łojasiewicz assumption on the objective function and some mild premises on stepsizes, the sequence produced by the proposed algorithm converges globally to a critical point of SVD-free RPCA models. Numerical simulations on synthetic and real datasets demonstrate the compelling performance of the proposed algorithm.

Distributed Proximal Gradient Algorithm for Nonconvex Optimization Over Time-Varying Networks

This paper studies the distributed non-convex optimization problem with non-smooth regularization, which has wide applications in decentralized learning, estimation and control. The objective function is the sum of local objective functions, which consist of differentiable (possibly non-convex) cost functions and non-smooth convex functions. This paper presents a distributed proximal gradient algorithm for the non-smooth non-convex optimization problem. Over time-varying multi-agent networks, the proposed algorithm updates local variable estimates with a constant step-size at the cost of multiple consensus steps, where the number of communication rounds increases over time. We prove that the generated local variables achieve consensus and converge to the set of critical points. Finally, we verify the efficiency of the proposed algorithm by numerical simulations.

On an iteratively reweighted linesearch based algorithm for nonconvex composite optimization

In this paper we propose a new algorithm for solving a class of nonsmooth nonconvex problems, which is obtained by combining the iteratively reweighted scheme with a finite number of forward–backward iterations based on a linesearch procedure. The new method overcomes some limitations of linesearch forward–backward methods, since it can be applied also to minimize functions containing terms that are both nonsmooth and nonconvex. Moreover, the combined scheme can take advantage of acceleration techniques consisting in suitable selection rules for the algorithm parameters. We develop the convergence analysis of the new method within the framework of the Kurdyka–Łojasiewicz property. Finally, we present the results of a numerical experience on microscopy image super resolution, showing that the performances of our method are comparable or superior to those of other algorithms designed for this specific application.

Gauss–Newton meets PANOC: A fast and globally convergent algorithm for nonlinear optimal control

PANOC is an algorithm for nonconvex optimization that has recently gained popularity in real-time control applications due to its fast, global convergence. The present work proposes a variant of PANOC that makes use of Gauss–Newton directions to accelerate the method. Furthermore, we show that when applied to optimal control problems, the computation of this Gauss–Newton step can be cast as a linear quadratic regulator (LQR) problem, allowing for an efficient solution through the Riccati recursion. Finally, we demonstrate that the proposed algorithm is more than twice as fast as the traditional L–BFGS variant of PANOC when applied to an optimal control benchmark problem, and that the performance scales favorably with increasing horizon length.

A symmetric splitting sequential quadratic optimization algorithm for two-block nonlinearly constrained nonconvex optimization

In this paper, a double-step-length symmetric splitting sequential quadratic optimization (DSL-SS-SQO) algorithm for solving two-block nonconvex optimization with nonlinear constraints is proposed. First, at each iteration, the idea of symmetric splitting is embedded into solving the quadratic optimization (QO) subproblem approximating the discussed problem. As a result, the QO subproblem is split into two small-scale QOs, which can generate two improved search directions for the two primal variables. Second, the augmented Lagrangian function is used as the merit function, and two step sizes are yielded by performing the Armijo line search along the two improved directions. Third, under mild conditions, the global convergence, strong convergence, iterative complexity, and Maratos effect of the DSL-SS-SQO algorithm are proven. Finally, some numerical results are reported, comparisons with the results obtained by the IPOPT solver are also provided, which preliminarily show that the proposed algorithm is promising.

Optimization conditions and decomposable algorithms for convertible nonconvex optimization

Optimization conditions and decomposable algorithms for convertible nonconvex optimization

Guaranteed Privacy of Distributed Nonconvex Optimization via Mixed-Monotone Functional Perturbations

In this letter, we introduce a new notion of guaranteed privacy for distributed nonconvex optimization algorithms. In particular, leveraging mixed-monotone inclusion functions, we propose a privacy-preserving mechanism which is based on deterministic, but unknown affine perturbations of the local objective functions. The design requires a robust optimization method to characterize the best accuracy that can be achieved by an optimal perturbation. This is used to guide the refinement of a guaranteed-private perturbation mechanism that can achieve a quantifiable accuracy via a theoretical upper bound that is independent of the chosen optimization algorithm. Finally, simulation results illustrate the accuracy-privacy trade-off and that our approach outperforms a benchmark differentially private distributed optimization algorithm in the literature.

Proximal gradient nonconvex optimization algorithm for the slice-based [formula omitted]-constrained convolutional dictionary learning

Convolutional dictionary learning (CDL) aims to learn a structured local convolutional dictionary and the sparse coefficient maps from the signals of various of interest, achieving better results than traditional patch-based dictionary learning in signal processing applications. Currently, most CDL methods are used to solve the ℓ1-constrained convex CDL problem using the patch-based, Fourier-based, or slice-based representation, which may lead to non-sparse coefficient maps, thus degrading the performance of applications. The slice-based representation via local processing has shown more beneficial than the Fourier-based representation for convex CDL problems. It can allow a different number of nonzeros for each spatial location of the signal according to the local complexity. However, it has not been used for the nonconvex CDL problem. In this paper, a slice-based ℓ0-constrained nonconvex CDL problem is proposed. It is the slice-based counterpart of the Fourier-based ℓ0-constrained nonconvex CDL problems and the nonconvex extension of the slice-based ℓ1-constrained convex CDL problems. A novel method named proximal gradient nonconvex optimization algorithm (PGNOA) is introduced. We prove that PGNOA can converge to a critical point. Extensive experiments are carried out on the benchmark data, and the results show that PGNOA is superior to the existing slice-based convex CDL methods and the Fourier-based nonconvex CDL methods in terms of the objective function value. The dictionaries learned from PGNOA are applied to image inpainting and image separation tasks. The experimental results demonstrate that PGNOA can perform better than other CDL methods.