Large-scale Convex Research Articles

Stabilized BB projection algorithm for large-scale convex constrained nonlinear monotone equations to signal and image processing problems

The Barzilai–Borwein (BB) method is a popular and efficient gradient method for solving large-scale unconstrained optimization problems. In general, it converges much faster than the steepest descent (Cauchy) method. However, it may not converge, even when the objective function is strongly convex. To overcome this, by virtue of bounding the distance between sequential iterates, [Burdakov et al. (2019)] proposed a stabilized BB method. In this paper, by combining the stabilized BB method with a projection approach, we develop a stabilized BB projection (SBBP) method to solve nonlinear monotone equations with convex constraints. SBBP does not involve computing matrices proving the usefulness for solving large-scale problems. Its global convergence and Q-linear convergence rate are also established. We report numerical experiments on recovering sparse signals and restoring blurred images arising from compressive sensing, demonstrating the usefulness of SBBP.

Stochastic Primal–Dual Hybrid Gradient Algorithm with Adaptive Step Sizes

In this work, we propose a new primal–dual algorithm with adaptive step sizes. The stochastic primal–dual hybrid gradient (SPDHG) algorithm with constant step sizes has become widely applied in large-scale convex optimization across many scientific fields due to its scalability. While the product of the primal and dual step sizes is subject to an upper-bound in order to ensure convergence, the selection of the ratio of the step sizes is critical in applications. Up-to-now there is no systematic and successful way of selecting the primal and dual step sizes for SPDHG. In this work, we propose a general class of adaptive SPDHG (A-SPDHG) algorithms and prove their convergence under weak assumptions. We also propose concrete parameters-updating strategies which satisfy the assumptions of our theory and thereby lead to convergent algorithms. Numerical examples on computed tomography demonstrate the effectiveness of the proposed schemes.

On the softplus penalty for large-scale convex optimization

We study a penalty reformulation of constrained convex optimization based on the softplus penalty function. For strongly convex objectives, we develop upper bounds on the objective value gap and the violation of constraints for the solutions to the penalty reformulations by analyzing the solution path of the reformulation with respect to the smoothness parameter. We use these upper bounds to analyze the complexity of applying gradient methods, which are advantageous when the number of constraints is large, to the reformulation.

Stochastic Localization Methods for Convex Discrete Optimization via Simulation

Solving Convex Discrete Optimization via Simulation via Stochastic Localization Algorithms Many decision-making problems in operations research and management science require the optimization of large-scale complex stochastic systems. For a number of applications, the objective function exhibits convexity in the discrete decision variables or the problem can be transformed into a convex one. In “Stochastic Localization Methods for Convex Discrete Optimization via Simulation,” Zhang, Zheng, and Lavaei propose provably efficient simulation-optimization algorithms for general large-scale convex discrete optimization via simulation problems. By utilizing the convex structure and the idea of localization and cutting-plane methods, the developed stochastic localization algorithms demonstrate a polynomial dependence on the dimension and scale of the decision space. In addition, the simulation cost is upper bounded by a value that is independent of the objective function. The stochastic localization methods also exhibit a superior numerical performance compared with existing algorithms.

Push-LSVRG-UP: Distributed Stochastic Optimization Over Unbalanced Directed Networks With Uncoordinated Triggered Probabilities

Distributed stochastic optimization, arising in the crossing and integration of traditional stochastic optimization, distributed computing and storage, and network science, has advantages of high efficiency and a low per-iteration computational complexity in resolving large-scale optimization problems. This paper concentrates on resolving a large-scale convex finite-sum optimization problem in a multi-agent system over unbalanced directed networks. To tackle this problem in an efficient way, a distributed consensus optimization algorithm, adopting the push-sum technique and a distributed loopless stochastic variance-reduced gradient (LSVRG) method with uncoordinated triggered probabilities, is developed and named Push-LSVRG-UP. Each agent under this algorithmic framework performs only local computation and communicates only with its neighbors without leaking their private information. The convergence analysis of Push-LSVRG-UP is relied on analyzing the contraction relationships between four error terms associated with the multi-agent system. Theoretical results provide an explicit feasible range of the constant step-size, a linear convergence rate, and an iteration complexity of Push-LSVRG-UP when achieving the globally optimal solution. It is shown that Push-LSVRG-UP achieves the superior characteristics of accelerated linear convergence, fewer storage costs, and a lower per-iteration computational complexity than most existing works. Meanwhile, the introduction of an uncoordinated probabilistic triggered mechanism allows Push-LSVRG-UP to facilitate the independence and flexibility of agents in computing local batch gradients. In simulations, the practicability and improved performance of Push-LSVRG-UP are manifested via resolving two distributed learning problems based on real-world datasets.

Bounding the detection efficiency threshold in Bell tests using multiple copies of the maximally entangled two-qubit state carried by a single pair of particles

In this paper, we investigate the critical efficiency of detectors to observe Bell nonlocality using multiple copies of the maximally entangled two-qubit state carried by a single pair of particles, such as hyperentangled states, and the product of Pauli measurements. It is known that in a Clauser-Horne-Shimony-Holt (CHSH) Bell test the symmetric detection efficiency of $82.84\%$ can be tolerated for the two-qubit maximally entangled state. We beat this enigmatic threshold by entangling two particles with multiple degrees of freedom. The obtained upper bounds of the symmetric detection efficiency thresholds are $80.86\%$, $73.99\%$ and $69.29\%$ for two, three and four copies of the two-qubit maximally entangled state, respectively. The number of measurements and outcomes in the respective cases are 4, 8 and 16. To find the improved thresholds, we use large-scale convex optimization tools, which allows us to significantly go beyond state-of-the-art results. The proof is exact up to three copies, while for four copies it is due to reliable numerical computations. Specifically, we used linear programming to obtain the two-copy threshold and the corresponding Bell inequality, and convex optimization based on Gilbert's algorithm for three and four copies of the two-qubit state. We show analytically that the symmetric detection efficiency threshold decays exponentially with the number of copies of the two-qubit state. Our techniques can also be applied to more general Bell nonlocality scenarios with more than two parties.

Distributed H∞ Method Design and Operation using Dis- tributed Computing

Control systems have two significant components: controllers and filters. Controllers are used to control the output while filters are used to estimate the internal state of a system from a series of noisy output measurements. The design of both require modelling, i.e the formulation of a mathematical model, taking into account all the dynamics of the system. There are several model based, “optimal” controllers and filters, such as the Kalman filter. However, when subjected to unaccounted errors, their performance suffers. “Robust” methods, such as H∞ methods for control and filtering, are capable of providing satisfactory performance, with respect to a performance parameter, even in the presence of noise. They are extensively used in sensitive applications such as spacecraft navigation, where a model cannot possibly account for all the errors. Their design requires convex optimization to minimize a convex function with respect to Linear Matrix Inequalities (LMI). In large scale convex optimization problems, centralized algorithms cannot work satisfactorily, due to slow convergence, and the need for a system with high performance and large memory. Distributed algorithms solve this practical constraint on convex optimization problems. However, current distributed algorithms are centralized and capable of convex optimization only within an infinite time horizon, offering sub-optimal results in finite time. In this project, we attempt to develop a distributed infinite-horizon algorithm which converges to an accurate value within a feasible number of iterations, and offers a speedup of at least 50% over traditional methods.

A hybrid algorithm for large-scale non-separable nonlinear multicommodity flow problems

We propose an approach for large-scale non-separable nonlinear multicommodity flow problems by solving a sequence of subproblems which can be addressed by commercial solvers. Using a combination of solution methods such as modified gradient projection, shortest path algorithm and golden section search, the approach can handle general problem instances, including those with (i) non-separable cost, (ii) objective function not available analytically as polynomial but are evaluated using black-boxes, and (iii) additional side constraints not of network flow types. Implemented as a toolbox in commercial solvers, it allows researchers and practitioners, currently conversant with linear instances, to easily manage large-scale convex instances as well. In this article, we compared the proposed algorithm with alternative approaches in the literature, covering both theory and large test cases. New test cases with non-separable convex costs and non-network flow side constraints are also presented and evaluated. The toolbox is available free for academic use upon request.

QPPAL: A Two-phase Proximal Augmented Lagrangian Method for High-dimensional Convex Quadratic Programming Problems

In this article, we aim to solve high-dimensional convex quadratic programming (QP) problems with a large number of quadratic terms, linear equality, and inequality constraints. To solve the targeted QP problem to a desired accuracy efficiently, we consider the restricted-Wolfe dual problem and develop a two-phase Proximal Augmented Lagrangian method (QPPAL), with Phase I to generate a reasonably good initial point to warm start Phase II to obtain an accurate solution efficiently. More specifically, in Phase I, based on the recently developed symmetric Gauss-Seidel (sGS) decomposition technique, we design a novel sGS-based semi-proximal augmented Lagrangian method for the purpose of finding a solution of low to medium accuracy. Then, in Phase II, a proximal augmented Lagrangian algorithm is proposed to obtain a more accurate solution efficiently. Extensive numerical results evaluating the performance of QPPAL against existing state-of-the-art solvers Gurobi, OSQP, and QPALM are presented to demonstrate the high efficiency and robustness of our proposed algorithm for solving various classes of large-scale convex QP problems. The MATLAB implementation of the software package QPPAL is available at https://blog.nus.edu.sg/mattohkc/softwares/qppal/ .

A Dimension Reduction Technique for Large-Scale Structured Sparse Optimization Problems with Application to Convex Clustering

In this paper, we propose a novel adaptive sieving (AS) technique and an enhanced AS (EAS) technique, which are solver independent and could accelerate optimization algorithms for solving large scale convex optimization problems with intrinsic structured sparsity. We establish the finite convergence property of the AS technique and the EAS technique with inexact solutions of the reduced subproblems. As an important application, we apply the AS technique and the EAS technique on the convex clustering model, which could accelerate the state-of-the-art algorithm SSNAL by more than 7 times and the algorithm ADMM by more than 14 times.

Secure Outsourcing of Large-Scale Convex Optimization Problem in Internet of Things

With the development of cloud computing and the advent of Internet of Things(IoT), outsourcing computation, as an important application of cloud computing, has been widely researched in the field of academic and industry. The convex optimization problem, as a most common mathematical problem, often appears in some machine learning algorithms and smart grid designs. However, the process of solving the convex optimization problem is very complicated and time-consuming. For some resource-constrained IoT devices, there are no enough computation resources and storage resources to deal with this problem. In this paper, we proposed an efficient and secure outsourcing algorithm for solving the large-scale convex optimization problem with equality constraints in IoT. Our proposed algorithm can not only reduce the computational complexity on the client side, but also protect the client’s sensitive data from being disclosed to the dishonest cloud server. In addition, the client can detect the malicious behavior from the cloud server with probability approximately 1. Finally, we give a theoretical analysis about correctness and security, and conduct experiments to show the feasibility of our proposed algorithm.

A faster stochastic alternating direction method for large scale convex composite problems

A faster stochastic alternating direction method for large scale convex composite problems

Distributed Proximal Splitting Algorithms with Rates and Acceleration

We analyze several generic proximal splitting algorithms well suited for large-scale convex nonsmooth optimization. We derive sublinear and linear convergence results with new rates on the function value suboptimality or distance to the solution, as well as new accelerated versions, using varying stepsizes. In addition, we propose distributed variants of these algorithms, which can be accelerated as well. While most existing results are ergodic, our nonergodic results significantly broaden our understanding of primal–dual optimization algorithms.

Fast Beampattern Synthesis Algorithm for Flexible Conformal Array

This letter first outlines a flexible conformal array geometric model on variable airborne wings, which can undergo complete shape change according to different combat situations, whereas the deformation may result in array beampattern distorted. In order to realize fast beampattern synthesis for a flexible conformal array, we propose a real-time beampattern optimization algorithm, where we split the protoproblem into two sub-problems based on the alternating direction method of multipliers (ADMM). In the first sub-problem, we transform the quadratic inequality constrained problem into a linear equality constrained problem, and then get the iterative update expression for the variables. Unlike the traditional interior point method, the simulation results demonstrate that the proposed algorithm can efficiently solve large-scale convex optimization problems and greatly reduce the computational complexity.

Adaptive uncertainty-weighted ADMM for distributed optimization

We present AUQ-ADMM, an adaptive uncertainty-weighted consensus ADMM method for solving large-scale convex optimization problems in a distributed manner. Our key contribution is a novel adaptive weighting scheme that empirically increases the progress made by consensus ADMM scheme and is attractive when using a large number of subproblems. The weights are related to the uncertainty associated with the solutions of each subproblem, and are efficiently computed using low-rank approximations. We show AUQ-ADMM provably converges and demonstrate its effectiveness on a series of machine learning applications, including elastic net regression, multinomial logistic regression, and support vector machines. We provide an implementation based on the PyTorch package.

A method with inertial extrapolation step for convex constrained monotone equations

In recent times, various algorithms have been incorporated with the inertial extrapolation step to speed up the convergence of the sequence generated by these algorithms. As far as we know, very few results exist regarding algorithms of the inertial derivative-free projection method for solving convex constrained monotone nonlinear equations. In this article, the convergence analysis of a derivative-free iterative algorithm (Liu and Feng in Numer. Algorithms 82(1):245–262, 2019) with an inertial extrapolation step for solving large scale convex constrained monotone nonlinear equations is studied. The proposed method generates a sufficient descent direction at each iteration. Under some mild assumptions, the global convergence of the sequence generated by the proposed method is established. Furthermore, some experimental results are presented to support the theoretical analysis of the proposed method.

An augmented Lagrangian method with constraint generation for shape-constrained convex regression problems

Shape-constrained convex regression problem deals with fitting a convex function to the observed data, where additional constraints are imposed, such as component-wise monotonicity and uniform Lipschitz continuity. This paper provides a unified framework for computing the least squares estimator of a multivariate shape-constrained convex regression function in $${\mathbb {R}}^d$$ . We prove that the least squares estimator is computable via solving an essentially constrained convex quadratic programming (QP) problem with $$(d+1)n$$ variables, $$n(n-1)$$ linear inequality constraints and n possibly non-polyhedral inequality constraints, where n is the number of data points. To efficiently solve the generally very large-scale convex QP, we design a proximal augmented Lagrangian method (proxALM) whose subproblems are solved by the semismooth Newton method. To further accelerate the computation when n is huge, we design a practical implementation of the constraint generation method such that each reduced problem is efficiently solved by our proposed proxALM. Comprehensive numerical experiments, including those in the pricing of basket options and estimation of production functions in economics, demonstrate that our proposed proxALM outperforms the state-of-the-art algorithms, and the proposed acceleration technique further shortens the computation time by a large margin.

A distributed algorithm for high-dimension convex quadratically constrained quadratic programs

We propose a Jacobi-style distributed algorithm to solve convex, quadratically constrained quadratic programs (QCQPs), which arise from a broad range of applications. While small to medium-sized convex QCQPs can be solved efficiently by interior-point algorithms, large-scale problems pose significant challenges to traditional algorithms that are mainly designed to be implemented on a single computing unit. The exploding volume of data (and hence, the problem size), however, may overwhelm any such units. In this paper, we propose a distributed algorithm for general, non-separable, large-scale convex QCQPs, using a novel idea of predictor-corrector primal-dual update with an adaptive step size. The algorithm enables distributed storage of data as well as parallel distributed computing. We establish the conditions for the proposed algorithm to converge to a global optimum, and implement our algorithm on a computer cluster with multiple nodes using Message Passing Interface (MPI). The numerical experiments are conducted on data sets of various scales from different applications, and the results show that our algorithm exhibits favorable scalability for solving large-scale problems.

Randomized Smoothing Variance Reduction Method for Large-Scale Non-smooth Convex Optimization

Randomized Smoothing Variance Reduction Method for Large-Scale Non-smooth Convex Optimization

Joint Estimation and Robustness Optimization

Many real-world optimization problems have input parameters estimated from data whose inherent imprecision can lead to fragile solutions that may impede desired objectives and/or render constraints infeasible. We propose a joint estimation and robustness optimization (JERO) framework to mitigate estimation uncertainty in optimization problems by seamlessly incorporating both the parameter estimation procedure and the optimization problem. Toward that end, we construct an uncertainty set that incorporates all of the data, and the size of the uncertainty set is based on how well the parameters are estimated from that data when using a particular estimation procedure: regressions, the least absolute shrinkage and selection operator, and maximum likelihood estimation (among others). The JERO model maximizes the uncertainty set’s size and so obtains solutions that—unlike those derived from models dedicated strictly to robust optimization—are immune to parameter perturbations that would violate constraints or lead to objective function values exceeding their desired levels. We describe several applications and provide explicit formulations of the JERO framework for a variety of estimation procedures. To solve the JERO models with exponential cones, we develop a second-order conic approximation that limits errors beyond an operating range; with this approach, we can use state-of-the-art second-order conic programming solvers to solve even large-scale convex optimization problems. This paper was accepted by J. George Shanthikumar, Management Science Special Section on Data-Driven Prescriptive Analytics.