Large-scale Convex Optimization Problems Research Articles

Distributed H∞ Method Design and Operation using Dis- tributed Computing

Control systems have two significant components: controllers and filters. Controllers are used to control the output while filters are used to estimate the internal state of a system from a series of noisy output measurements. The design of both require modelling, i.e the formulation of a mathematical model, taking into account all the dynamics of the system. There are several model based, “optimal” controllers and filters, such as the Kalman filter. However, when subjected to unaccounted errors, their performance suffers. “Robust” methods, such as H∞ methods for control and filtering, are capable of providing satisfactory performance, with respect to a performance parameter, even in the presence of noise. They are extensively used in sensitive applications such as spacecraft navigation, where a model cannot possibly account for all the errors. Their design requires convex optimization to minimize a convex function with respect to Linear Matrix Inequalities (LMI). In large scale convex optimization problems, centralized algorithms cannot work satisfactorily, due to slow convergence, and the need for a system with high performance and large memory. Distributed algorithms solve this practical constraint on convex optimization problems. However, current distributed algorithms are centralized and capable of convex optimization only within an infinite time horizon, offering sub-optimal results in finite time. In this project, we attempt to develop a distributed infinite-horizon algorithm which converges to an accurate value within a feasible number of iterations, and offers a speedup of at least 50% over traditional methods.

A Dimension Reduction Technique for Large-Scale Structured Sparse Optimization Problems with Application to Convex Clustering

In this paper, we propose a novel adaptive sieving (AS) technique and an enhanced AS (EAS) technique, which are solver independent and could accelerate optimization algorithms for solving large scale convex optimization problems with intrinsic structured sparsity. We establish the finite convergence property of the AS technique and the EAS technique with inexact solutions of the reduced subproblems. As an important application, we apply the AS technique and the EAS technique on the convex clustering model, which could accelerate the state-of-the-art algorithm SSNAL by more than 7 times and the algorithm ADMM by more than 14 times.

Secure Outsourcing of Large-Scale Convex Optimization Problem in Internet of Things

With the development of cloud computing and the advent of Internet of Things(IoT), outsourcing computation, as an important application of cloud computing, has been widely researched in the field of academic and industry. The convex optimization problem, as a most common mathematical problem, often appears in some machine learning algorithms and smart grid designs. However, the process of solving the convex optimization problem is very complicated and time-consuming. For some resource-constrained IoT devices, there are no enough computation resources and storage resources to deal with this problem. In this paper, we proposed an efficient and secure outsourcing algorithm for solving the large-scale convex optimization problem with equality constraints in IoT. Our proposed algorithm can not only reduce the computational complexity on the client side, but also protect the client’s sensitive data from being disclosed to the dishonest cloud server. In addition, the client can detect the malicious behavior from the cloud server with probability approximately 1. Finally, we give a theoretical analysis about correctness and security, and conduct experiments to show the feasibility of our proposed algorithm.

Fast Beampattern Synthesis Algorithm for Flexible Conformal Array

This letter first outlines a flexible conformal array geometric model on variable airborne wings, which can undergo complete shape change according to different combat situations, whereas the deformation may result in array beampattern distorted. In order to realize fast beampattern synthesis for a flexible conformal array, we propose a real-time beampattern optimization algorithm, where we split the protoproblem into two sub-problems based on the alternating direction method of multipliers (ADMM). In the first sub-problem, we transform the quadratic inequality constrained problem into a linear equality constrained problem, and then get the iterative update expression for the variables. Unlike the traditional interior point method, the simulation results demonstrate that the proposed algorithm can efficiently solve large-scale convex optimization problems and greatly reduce the computational complexity.

Adaptive uncertainty-weighted ADMM for distributed optimization

We present AUQ-ADMM, an adaptive uncertainty-weighted consensus ADMM method for solving large-scale convex optimization problems in a distributed manner. Our key contribution is a novel adaptive weighting scheme that empirically increases the progress made by consensus ADMM scheme and is attractive when using a large number of subproblems. The weights are related to the uncertainty associated with the solutions of each subproblem, and are efficiently computed using low-rank approximations. We show AUQ-ADMM provably converges and demonstrate its effectiveness on a series of machine learning applications, including elastic net regression, multinomial logistic regression, and support vector machines. We provide an implementation based on the PyTorch package.

Joint Estimation and Robustness Optimization

Many real-world optimization problems have input parameters estimated from data whose inherent imprecision can lead to fragile solutions that may impede desired objectives and/or render constraints infeasible. We propose a joint estimation and robustness optimization (JERO) framework to mitigate estimation uncertainty in optimization problems by seamlessly incorporating both the parameter estimation procedure and the optimization problem. Toward that end, we construct an uncertainty set that incorporates all of the data, and the size of the uncertainty set is based on how well the parameters are estimated from that data when using a particular estimation procedure: regressions, the least absolute shrinkage and selection operator, and maximum likelihood estimation (among others). The JERO model maximizes the uncertainty set’s size and so obtains solutions that—unlike those derived from models dedicated strictly to robust optimization—are immune to parameter perturbations that would violate constraints or lead to objective function values exceeding their desired levels. We describe several applications and provide explicit formulations of the JERO framework for a variety of estimation procedures. To solve the JERO models with exponential cones, we develop a second-order conic approximation that limits errors beyond an operating range; with this approach, we can use state-of-the-art second-order conic programming solvers to solve even large-scale convex optimization problems. This paper was accepted by J. George Shanthikumar, Management Science Special Section on Data-Driven Prescriptive Analytics.

Memory-Efficient Structured Convex Optimization via Extreme Point Sampling

Memory is a key computational bottleneck when solving large-scale convex optimization problems, such as semidefinite programs (SDPs). In this paper, we focus on the regime in which storing an $n\ti...

Decentralized stochastic optimization algorithms using uncoordinated step-sizes over unbalanced directed networks

Decentralized stochastic gradient methods play significant roles in large-scale optimization that finds many practical applications in signal processing, machine learning, and coordinated control. In this short communication, we focus on studying large-scale convex optimization problems over multi-agent systems, where the mutual goal of agents in the network is to optimize a global objective function expressed as a sum of local objective functions. Each agent in the system uses only local computation and communication in the overall process without leaking their private information. We first introduce both the local stochastic averaging gradient method and the local stochastic variance-reduced gradient method into decentralized convex optimization (DCO) over unbalanced directed networks. Then, in order to increase the autonomy and flexibility of the agents, uncoordinated step-sizes are considered. Based on the well-known variance-reduced stochastic gradient methods and uncoordinated step-sizes, we propose two fully decentralized algorithms named SAGA-UDN and SVRG-UDN for DCO over unbalanced directed networks, respectively. Finally, both SAGA-UDN and SVRG-UDN are numerically studied under various network parameters and objective functions. Some real-world data sets are used in simulations to demonstrate the improved performance of SAGA-UDN and SVRG-UDN.

Non-Monotone Projected Gradient Method in Linear Elasticity Contact Problems with Given Friction

We are focusing on the algorithms for solving the large-scale convex optimization problem in linear elasticity contact problems discretized by Finite Element method (FEM). The unknowns of the problem are the displacements of the FEM nodes, the corresponding objective function is defined as a convex quadratic function with symmetric positive definite stiffness matrix and additional non-linear term representing the friction in contact. The feasible set constraints the displacement subject to non-penetration conditions. The dual formulation of this optimization problem is well-known as a Quadratic Programming (QP) problem and can be considered as a most basic non-linear optimization problem. Understanding these problems and the development of efficient algorithms for solving them play the crucial role in the large-scale problems in practical applications. We shortly review the theory and examine the behavior and the efficiency of Spectral Projected Gradient method modified for QP problems (SPG-QP) on the solution of a toy example in MATLAB environment.

A Generalized Accelerated Composite Gradient Method: Uniting Nesterov's Fast Gradient Method and FISTA

The most popular first-order accelerated black-box methods for solving large-scale convex optimization problems are the Fast Gradient Method (FGM) and the Fast Iterative Shrinkage Thresholding Algorithm (FISTA). FGM requires that the objective be finite and differentiable with known gradient Lipschitz constant. FISTA is applicable to the more broad class of composite objectives and is equipped with a line-search procedure for estimating the Lipschitz constant. Nonetheless, FISTA cannot increase the step size and is unable to take advantage of strong convexity. FGM and FISTA are very similar in form. Despite this, they appear to have vastly differing convergence analyses. In this work we generalize the previously introduced augmented estimate sequence framework as well as the related notion of the gap sequence. We showcase the flexibility of our tools by constructing a Generalized Accelerated Composite Gradient Method, that unites FGM and FISTA, along with their most popular variants. The Lyapunov property of the generalized gap sequence used in deriving our method implies that both FGM and FISTA are amenable to a Lyapunov analysis, common among optimization algorithms. We further showcase the flexibility of our tools by endowing our method with monotonically decreasing objective function values alongside a versatile line-search procedure. By simultaneously incorporating the strengths of FGM and FISTA, our method is able to surpass both in terms of robustness and usability. We support our findings with simulation results on an extensive benchmark of composite problems. Our experiments show that monotonicity has a stabilizing effect on convergence and challenge the notion present in the literature that for strongly convex objectives, accelerated proximal schemes can be reduced to fixed momentum methods.

Feature selection for probabilistic load forecasting via sparse penalized quantile regression

Probabilistic load forecasting (PLF) is able to present the uncertainty information of the future loads. It is the basis of stochastic power system planning and operation. Recent works on PLF mainly focus on how to develop and combine forecasting models, while the feature selection issue has not been thoroughly investigated for PLF. This paper fills the gap by proposing a feature selection method for PLF via sparse L_1-norm penalized quantile regression. It can be viewed as an extension from point forecasting-based feature selection to probabilistic forecasting-based feature selection. Since both the number of training samples and the number of features to be selected are very large, the feature selection process is casted as a large-scale convex optimization problem. The alternating direction method of multipliers is applied to solve the problem in an efficient manner. We conduct case studies on the open datasets of ten areas. Numerical results show that the proposed feature selection method can improve the performance of the probabilistic forecasting and outperforms traditional least absolute shrinkage and selection operator method.

Joint Estimation and Robustness Optimization

Many real-world optimization problems have input parameters estimated from data whose inherent imprecision can lead to fragile solutions that may impede desired objectives and/or render constraints infeasible. We propose a joint estimation and robustness optimization (JERO) framework to mitigate estimation uncertainty in optimization problems by seamlessly incorporating both the parameter estimation procedure and the optimization problem. Toward that end, we construct an uncertainty set that incorporates all of the data, where the size of the uncertainty set is based on how well the parameters would be estimated from that data when using a particular estimation procedure: regressions, the least absolute shrinkage and selection operator, and maximum likelihood estimation (among others). The JERO model maximizes the uncertainty set's size and so obtains solutions that---unlike those derived from models dedicated strictly to robust optimization---are immune to parameter perturbations that would violate constraints or lead to objective function values exceeding their desired levels. We describe several applications and provide explicit formulations of the JERO framework for a variety of estimation procedures. To solve the JERO models with exponential cones, we develop a second-order conic approximation that limits errors beyond an operating range; with this approach, we can use state-of-the-art SOCP solvers to solve even large-scale convex optimization problems. Finally, we apply the JERO model to a case study, thereby addressing a health insurance reimbursement problem with the aim of improving patient flow in the healthcare system while hedging against estimation errors.

A Fast Gradient Method for Nonnegative Sparse Regression With Self-Dictionary.

A nonnegative matrix factorization (NMF) can be computed efficiently under the separability assumption, which asserts that all the columns of the given input data matrix belong to the cone generated by a (small) subset of them. The provably most robust methods to identify these conic basis columns are based on nonnegative sparse regression and self-dictionaries, and require the solution of large-scale convex optimization problems. In this paper, we study a particular nonnegative sparse regression model with self-dictionary. As opposed to previously proposed models, this model yields a smooth optimization problem, where the sparsity is enforced through linear constraints. We show that the Euclidean projection on the polyhedron defined by these constraints can be computed efficiently, and propose a fast gradient method to solve our model. We compare our algorithm with several state-of-the-art methods on synthetic data sets and real-world hyperspectral images.A nonnegative matrix factorization (NMF) can be computed efficiently under the separability assumption, which asserts that all the columns of the given input data matrix belong to the cone generated by a (small) subset of them. The provably most robust methods to identify these conic basis columns are based on nonnegative sparse regression and self-dictionaries, and require the solution of large-scale convex optimization problems. In this paper, we study a particular nonnegative sparse regression model with self-dictionary. As opposed to previously proposed models, this model yields a smooth optimization problem, where the sparsity is enforced through linear constraints. We show that the Euclidean projection on the polyhedron defined by these constraints can be computed efficiently, and propose a fast gradient method to solve our model. We compare our algorithm with several state-of-the-art methods on synthetic data sets and real-world hyperspectral images.

Sparse Subspace Clustering by Learning Approximation ℓ0 Codes

Subspace clustering has been widely applied to detect meaningful clusters in high-dimensional data spaces. A main challenge in subspace clustering is to quickly calculate a "good" affinity matrix. ℓ0, ℓ1, ℓ2 or nuclear norm regularization is used to construct the affinity matrix in many subspace clustering methods because of their theoretical guarantees and empirical success. However, they suffer from the following problems: (1) ℓ2 and nuclear norm regularization require very strong assumptions to guarantee a subspace-preserving affinity; (2) although ℓ1 regularization can be guaranteed to give a subspace-preserving affinity under certain conditions, it needs more time to solve a large-scale convex optimization problem; (3) ℓ0 regularization can yield a tradeoff between computationally efficient and subspace-preserving affinity by using the orthogonal matching pursuit (OMP) algorithm, but this still takes more time to search the solution in OMP when the number of data points is large. In order to overcome these problems, we first propose a learned OMP (LOMP) algorithm to learn a single hidden neural network (SHNN) to fast approximate the ℓ0code. We then exploit a sparse subspace clustering method based on ℓ0 code which is fast computed by SHNN. Two sufficient conditions are presented to guarantee that our method can give a subspace-preserving affinity. Experiments on handwritten digit and face clustering show that our method not only quickly computes the ℓ0 code, but also outperforms the relevant subspace clustering methods in clustering results. In particular, our method achieves the state-of-the-art clustering accuracy (94.32%) on MNIST.

Joint Energy Management Strategy for Geo-Distributed Data Centers and Electric Vehicles in Smart Grid Environment

According to some forecast models, electric vehicles (EVs) are expected to have a high level of penetration in the coming decades. To support the daily operation of EVs, charging is necessary. Generally, there are three main charging scenes in the future, i.e., charging at workplace, charging at home, and charging at commercial station. For a data center operator, the synchronous EV charging of the employees at the workplace during the working hours would incur an additional large demand charge. To avoid/reduce such demand charge, we investigate a joint energy management problem for geo-distributed data centers and EVs of the employees in this paper. Specifically, we intend to minimize the cost of a data center operator by jointly scheduling data center workload and EV charging under the given power limits, where the cost consists of electricity bill, revenue loss associated with workloads, and battery depreciation cost. We first formulate a total cost minimization problem with the consideration of heterogeneous demands of EVs and the given power limits. Since the formulated problem is a large-scale convex optimization problem with temporally-coupled and spatially-coupled constraints, we propose a distributed algorithm to solve it. Based on the proposed algorithm, we design a joint energy management strategy for geo-distributed data centers and EVs. Simulation results based on real-world traces show that the proposed strategy could reduce the cost for the data center operator by up to 5.324%.

Large-scale optimization with the primal-dual column generation method

The primal-dual column generation method (PDCGM) is a general-purpose column generation technique that relies on the primal-dual interior point method to solve the restricted master problems. The use of this interior point method variant allows to obtain suboptimal and well-centered dual solutions which naturally stabilizes the column generation process. As recently presented in the literature, reductions in the number of calls to the oracle and in the CPU times are typically observed when compared to the standard column generation, which relies on extreme optimal dual solutions. However, these results are based on relatively small problems obtained from linear relaxations of combinatorial applications. In this paper, we investigate the behaviour of the PDCGM in a broader context, namely when solving large-scale convex optimization problems. We have selected applications that arise in important real-life contexts such as data analysis (multiple kernel learning problem), decision-making under uncertainty (two-stage stochastic programming problems) and telecommunication and transportation networks (multicommodity network flow problem). In the numerical experiments, we use publicly available benchmark instances to compare the performance of the PDCGM against recent results for different methods presented in the literature, which were the best available results to date. The analysis of these results suggests that the PDCGM offers an attractive alternative over specialized methods since it remains competitive in terms of number of iterations and CPU times even for large-scale optimization problems.

Large-Scale Convex Optimization for Dense Wireless Cooperative Networks

Convex optimization is a powerful tool for resource allocation and signal processing in wireless networks. As the network density is expected to drastically increase in order to accommodate the exponentially growing mobile data traffic, performance optimization problems are entering a new era characterized by a high dimension and/or a large number of constraints, which poses significant design and computational challenges. In this paper, we present a novel two-stage approach to solve large-scale convex optimization problems for dense wireless cooperative networks, which can effectively detect infeasibility and enjoy modeling flexibility. In the proposed approach, the original large-scale convex problem is transformed into a standard cone programming form in the first stage via matrix stuffing, which only needs to copy the problem parameters such as channel state information (CSI) and quality-of-service (QoS) requirements to the prestored structure of the standard form. The capability of yielding infeasibility certificates and enabling parallel computing is achieved by solving the homogeneous self-dual embedding of the primal-dual pair of the standard form. In the solving stage, the operator splitting method, namely, the alternating direction method of multipliers (ADMM), is adopted to solve the large-scale homogeneous self-dual embedding. Compared with second-order methods, ADMM can solve large-scale problems in parallel with modest accuracy within a reasonable amount of time. Simulation results will demonstrate the speedup, scalability, and reliability of the proposed framework compared with the state-of-the-art modeling frameworks and solvers.

Large-scale convex optimization for ultra-dense cloud-RAN

The heterogeneous cloud radio access network (Cloud-RAN) provides a revolutionary way to densify radio access networks. It enables centralized coordination and signal processing for efficient interference management and flexible network adaptation. Thus it can resolve the main challenges for next-generation wireless networks, including higher energy efficiency and spectral efficiency, higher cost efficiency, scalable connectivity, and low latency. In this article we will provide an algorithmic approach to the new design challenges for the dense heterogeneous Cloud-RAN based on convex optimization. As problem sizes scale up with the network size, we will demonstrate that it is critical to take unique structures of design problems and inherent characteristics of wireless channels into consideration, while convex optimization will serve as a powerful tool for such purposes. Network power minimization and channel state information acquisition will be used as two typical examples to demonstrate the effectiveness of convex optimization methods. Then we will present a twostage framework to solve general large-scale convex optimization problems, which is amenable to parallel implementation in the cloud data center.

On Solving Large-Scale Polynomial Convex Problems by Randomized First-Order Algorithms

One of the most attractive recent approaches to processing well-structured large-scale convex optimization problems is based on smooth convex-concave saddle point reformulation of the problem of interest and solving the resulting problem by a fast first order saddle point method utilizing smoothness of the saddle point cost function. In this paper, we demonstrate that when the saddle point cost function is polynomial, the precise gradients of the cost function required by deterministic first order saddle point algorithms and becoming prohibitively computationally expensive in the extremely large-scale case, can be replaced with incomparably cheaper computationally unbiased random estimates of the gradients. We show that for large-scale problems with favorable geometry, this randomization accelerates, progressively as the sizes of the problem grow, the solution process. This extends significantly previous results on acceleration by randomization, which, to the best of our knowledge, dealt solely with bilinear saddle point problems. We illustrate our theoretical findings by instructive and encouraging numerical experiments.

Restricted strong convexity and its applications to convergence analysis of gradient-type methods in convex optimization

A great deal of interest of solving large-scale convex optimization problems has, recently, turned to gradient method and its variants. To ensure rates of linear convergence, current theory regularly assumes that the objective functions are strongly convex. This paper goes beyond the traditional wisdom by studying a strictly weaker concept than the strong convexity, named restricted strongly convexity which was recently proposed and can be satisfied by a much broader class of functions. Utilizing the restricted strong convexity, we derive rates of linear convergence for (in)exact gradient-type methods. Besides, we obtain two by-products: (1) we rederive rates of linear convergence of inexact gradient method for a class of structured smooth convex optimizations; (2) we improve the rate of linear convergence for the linearized Bregman algorithm.