Constrained Convex Optimization Problem Research Articles

An inexact proximal generalized alternating direction method of multipliers

This paper proposes and analyzes an inexact variant of the proximal generalized alternating direction method of multipliers (ADMM) for solving separable linearly constrained convex optimization problems. In this variant, the first subproblem is approximately solved using a relative error condition whereas the second one is assumed to be easy to solve. In many ADMM applications, one of the subproblems has a closed-form solution; for instance, $$\ell _1$$ regularized convex composite optimization problems. The proposed method possesses iteration-complexity bounds similar to its exact version. More specifically, it is shown that, for a given tolerance $$\rho >0$$ , an approximate solution of the Lagrangian system associated to the problem under consideration is obtained in at most $$\mathcal {O}(1/\rho ^2)$$ (resp. $$\mathcal {O}(1/\rho )$$ in the ergodic case) iterations. Numerical experiments are presented to illustrate the performance of the proposed scheme.

A Fixed-Point Subgradient Splitting Method for Solving Constrained Convex Optimization Problems

In this work, we consider a bilevel optimization problem consisting of the minimizing sum of two convex functions in which one of them is a composition of a convex function and a nonzero linear transformation subject to the set of all feasible points represented in the form of common fixed-point sets of nonlinear operators. To find an optimal solution to the problem, we present a fixed-point subgradient splitting method and analyze convergence properties of the proposed method provided that some additional assumptions are imposed. We investigate the solving of some well known problems by using the proposed method. Finally, we present some numerical experiments for showing the effectiveness of the obtained theoretical result.

A unitary distributed subgradient method for multi-agent optimization with different coupling sources

Distributed optimization techniques offer high quality solutions to various engineering problems, such as resource allocation and distributed estimation and control. In this work, we first consider distributed convex constrained optimization problems where the objective function is encoded by multiple local and possibly nonsmooth objectives privately held by a group of agents, and propose a distributed subgradient method with double averaging (abbreviated as DSA2) that only requires peer-to-peer communication and local computation to solve the global problem. The algorithmic framework builds on dual methods and dynamic average consensus; the sequence of test points is formed by iteratively minimizing a local dual model of the overall objective where the coefficients, i.e., approximated subgradients of the objective, are supplied by the dynamic average consensus scheme. We theoretically show that DSA2 enjoys non-ergodic convergence properties, i.e., the local minimizing sequence itself is convergent, a distinct feature that cannot be found in existing results. Specifically, we establish a convergence rate of O(1t) in terms of objective function error. Then, extensions are made to tackle distributed optimization problems with coupled functional constraints by combining DSA2 and dual decomposition. This is made possible by Lagrangian relaxation that transforms the coupling in constraints of the primal problem into that in cost functions of the dual, thus allowing us to solve the dual problem via DSA2. Both the dual objective error and the quadratic penalty for the coupled constraint are proved to converge at a rate of O(1t), and the primal objective error asymptotically vanishes. Numerical experiments and comparisons are conducted to illustrate the advantage of the proposed algorithms and validate our theoretical findings.

Nesterov Acceleration for Equality-Constrained Convex Optimization via Continuously Differentiable Penalty Functions

We propose a framework to use Nesterov's accelerated method for constrained convex optimization problems. Our approach consists of first reformulating the original problem as an unconstrained optimization problem using a continuously differentiable exact penalty function. This reformulation is based on replacing the Lagrange multipliers in the augmented Lagrangian of the original problem by Lagrange multiplier functions. The expressions of these Lagrange multiplier functions, which depend upon the gradients of the objective function and the constraints, can make the unconstrained penalty function non-convex in general even if the original problem is convex. We establish sufficient conditions on the objective function and the constraints of the original problem under which the unconstrained penalty function is convex. This enables us to use Nesterov's accelerated gradient method for unconstrained convex optimization and achieve a guaranteed rate of convergence which is better than the state-of-the-art first-order algorithms for constrained convex optimization. Simulations illustrate our results.

Alternating direction method of multipliers with variable metric indefinite proximal terms for convex optimization

<p style='text-indent:20px;'>This paper studies a proximal alternating direction method of multipliers (ADMM) with variable metric indefinite proximal terms for linearly constrained convex optimization problems. The proximal ADMM plays an important role in many application areas, since the subproblems of the method are easy to solve. Recently, it is reported that the proximal ADMM with a certain fixed indefinite proximal term is faster than that with a positive semidefinite term, and still has the global convergence property. On the other hand, Gu and Yamashita studied a variable metric semidefinite proximal ADMM whose proximal term is generated by the BFGS update. They reported that a slightly indefinite matrix also makes the algorithm work well in their numerical experiments. Motivated by this fact, we consider a variable metric indefinite proximal ADMM, and give sufficient conditions on the proximal terms for the global convergence. Moreover, based on the BFGS update, we propose a new indefinite proximal term which can satisfy the conditions for the global convergence. Experiments on several datasets demonstrated that our proposed variable metric indefinite proximal ADMM outperforms most of the comparison proximal ADMMs.

Random Gradient-Free Optimization for Multiagent Systems With Communication Noises Under a Time-Varying Weight Balanced Digraph

In this paper, we focus on a constrained convex optimization problem of multiagent systems under a time-varying topology. In such topology, it is not only B-strongly connected, but the communication noises are also existent. Each agent has access to its local cost function, which is a nonsmooth function. A gradient-free random protocol is come up with minimizing a sum of cost functions of all agents, which are projected to local constraint sets. First, considering the stochastic disturbances in the communication channels among agents, the upper bounds of disagreement estimate of agents’ states are obtained. Second, a sufficient condition on choosing step sizes and smoothing parameters is derived to guarantee that all agents almost surely converge to the stationary optimal point. At last, a numerical example and a comparison are provided to illustrate the feasibility of the random gradient-free algorithm.

Limit Behavior and the Role of Augmentation in Projected Saddle Flows for Convex Optimization

In this paper, we study the stability and convergence of continuous-time Lagrangian saddle flows to solutions of a convex constrained optimization problem. Convergence of these flows is well-known when the underlying saddle function is either strictly convex in the primal or strictly concave in the dual variables. In this paper, we show convergence under non-strict convexity when a simple, unilateral augmentation term is added. For this purpose, we establish a novel, non-trivial characterization of the limit set of saddle-flow trajectories that allows us to preclude limit cycles. With our presentation we try to unify several existing problem formulations as a projected dynamical system that allows projection of both the primal and dual variables, thus complementing results available in the recent literature.

An adaptive primal-dual framework for nonsmooth convex minimization

We propose a new self-adaptive and double-loop smoothing algorithm to solve composite, nonsmooth, and constrained convex optimization problems. Our algorithm is based on Nesterov’s smoothing technique via general Bregman distance functions. It self-adaptively selects the number of iterations in the inner loop to achieve a desired complexity bound without requiring to set the accuracy a priori as in variants of augmented Lagrangian methods (ALM). We prove $$\mathcal {O}\left( \frac{1}{k}\right) $$ -convergence rate on the last iterate of the outer sequence for both unconstrained and constrained settings in contrast to ergodic rates which are common in ALM as well as alternating direction method-of-multipliers literature. Compared to existing inexact ALM or quadratic penalty methods, our analysis does not rely on the worst-case bounds of the subproblem solved by the inner loop. Therefore, our algorithm can be viewed as a restarting technique applied to the ASGARD method in Tran-Dinh et al. (SIAM J Optim 28(1):96–134, 2018) but with rigorous theoretical guarantees or as an inexact ALM with explicit inner loop termination rules and adaptive parameters. Our algorithm only requires to initialize the parameters once, and automatically updates them during the iteration process without tuning. We illustrate the superiority of our methods via several examples as compared to the state-of-the-art.

Two New Customized Proximal Point Algorithms Without Relaxation for Linearly Constrained Convex Optimization

In this paper, we propose a new customized proximal point algorithm for linearly constrained convex optimization problem, and further extend the proposed method to separable convex optimization problem. Unlike the existing customized proximal point algorithms, the proposed algorithms do not involve relaxation step, but still ensure the convergence. We obtain the particular iteration schemes and the unified variational inequality perspective. The global convergence and $${\mathcal {O}}(1/k)$$-convergence rate of the proposed methods are investigated under some mild assumptions. Numerical experiments show that compared to some state-of-the-art methods, the proposed methods are effective.

Fast and Effective Learning for Fuzzy Cognitive Maps: A Method Based on Solving Constrained Convex Optimization Problems

The learning of fuzzy cognitive maps (FCMs) is a timely issue pursued by numerous researchers. Many learning methods, such as population-based algorithms and some hybrid algorithms, have been developed and applied to many fields resulting in better performance. However, those learning algorithms also exhibit obvious limitations: some of them either are difficult to handle the learning problem of large-scale FCM or extremely time-consuming with high computational overload. Furthermore, the learning problem of FCM with noisy data is rarely considered in existing algorithms. In this article, a fast and efficient method for learning FCM is proposed. It first transforms the learning problem of the FCM into a convex optimization problem with constraints, and then, the classic interior-point methods are invoked to solve the optimization problem to obtain the optimized weight matrix of the FCM. A series of experiments involving synthetic data, noisy synthetic data, real data, and publicly available data with noise are considered to demonstrate that the proposed method can rapidly and efficiently learn small-scale and large-scale FCMs and deal with the problem of learning FCM from noisy historical data.

Saddle-Flow Dynamics for Distributed Feedback-Based Optimization

This letter develops a distributed saddle-flow algorithm to regulate the output of a networked system—modeled as static linear map—to the solution of a constrained convex optimization problem. The algorithm is “feedback-based”, in the sense that measurements of the network output are leveraged in the saddle-flow updates to avoid a complete (oracle-based) knowledge of the network map. In the distributed architecture, each actuator has access to only a subset of measurements; nevertheless, supported by a connected communication graph, a distributed protocol is implemented to achieve consensus on pertinent dual variables associated with network-level output constraints and, therefore, on the solution of the constrained problem. Using a LaSalle argument, we show that under an easily satisfiable linear matrix inequality condition the proposed algorithm converges to an optimal primal–dual solution. We demonstrate the effectiveness of the proposed method in a voltage regulation problem for power systems with high penetration of renewable generation.

A Peaceman–Rachford Splitting Method with Monotone Plus Skew-Symmetric Splitting for Nonlinear Saddle Point Problems

This paper is devoted to solving the linearly constrained convex optimization problems by Peaceman–Rachford splitting method with monotone plus skew-symmetric splitting on KKT operators. This approach generalizes the Hermitian and skew-Hermitian splitting method, an unconditionally convergent algorithm for non-Hermitian positive definite linear systems, to the nonlinear scenario. The convergence of the proposed algorithm is guaranteed under some mild assumptions, e.g., the strict convexity on objective functions and the consistency on constraints, even though the Lions–Mercier property is not fulfilled. In addition, we explore an inexact version of the proposed algorithm, which allows solving the subproblems approximately with some inexactness criteria. Numerical simulations on an image restoration problem demonstrate the compelling performance of the proposed algorithm.

A hybrid descent method for solving a convex constrained optimization problem with applications

The purpose of this paper is to introduce a hybrid descent algorithm for finding a common point in fixed‐point sets of quasi‐nonexpansive mappings and solution sets of variational inequality problems. In the framework of Hilbert spaces, the strong convergence of the hybrid descent algorithm is established. Numerical experiments for the bandwidth allocation, which demonstrate the effectiveness, performance, and convergence of the proposed algorithm, are provided.

Low-Complexity Bit Allocation for RSS Target Localization

The overall localization error performance for a sensor network employing quantized received signal strength (RSS) measurements largely depends on the number of bits used and their distribution. In this paper, we consider a rate-constrained bit allocation problem in which multiple points of interest (PoIs) are assigned possibly non-uniform error requirements. The main objective is to allocate a number of bits in order to satisfy error requirements as best as possible in a squared error sense. Due to the computational complexity of optimally solving the allocation problem, two novel low complexity allocation algorithms are proposed. The main idea behind these algorithms is to relate localization errors to corresponding bit density requirements. In the first algorithm, bit allocation is formulated as a sequential constrained convex optimization problem which can be solved with efficiency using available convex solvers. The second algorithm is a greedy version of the first algorithm in which sensors are selected for allocation based on the bit density differences at their neighboring PoIs. Numerical experiments show the efficacy of the proposed algorithms in comparison to other available algorithms in terms of overall error, coverage, and computational complexity.

Incremental and Parallel Machine Learning Algorithms With Automated Learning Rate Adjustments.

The existing machine learning algorithms for minimizing the convex function over a closed convex set suffer from slow convergence because their learning rates must be determined before running them. This paper proposes two machine learning algorithms incorporating the line search method, which automatically and algorithmically finds appropriate learning rates at run-time. One algorithm is based on the incremental subgradient algorithm, which sequentially and cyclically uses each of the parts of the objective function; the other is based on the parallel subgradient algorithm, which uses parts independently in parallel. These algorithms can be applied to constrained nonsmooth convex optimization problems appearing in tasks of learning support vector machines without adjusting the learning rates precisely. The proposed line search method can determine learning rates to satisfy weaker conditions than the ones used in the existing machine learning algorithms. This implies that the two algorithms are generalizations of the existing incremental and parallel subgradient algorithms for solving constrained nonsmooth convex optimization problems. We show that they generate sequences that converge to a solution of the constrained nonsmooth convex optimization problem under certain conditions. The main contribution of this paper is the provision of three kinds of experiment showing that the two algorithms can solve concrete experimental problems faster than the existing algorithms. First, we show that the proposed algorithms have performance advantages over the existing ones in solving a test problem. Second, we compare the proposed algorithms with a different algorithm Pegasos, which is designed to learn with a support vector machine efficiently, in terms of prediction accuracy, value of the objective function, and computational time. Finally, we use one of our algorithms to train a multilayer neural network and discuss its applicability to deep learning.

Stable Conical Regularization by Constructible Dilating Cones with an Application to $L^{p}$-constrained Optimization Problems

We study a convex constrained optimization problem that suffers from the lack of Slater-type constraint qualification. By employing a constructible representation of the constraint cone, we devise a new family of dilating cones and use it to introduce a family of regularized problems. We establish novel stability estimates for the regularized problems in terms of the regularization parameter. To show the feasibility and efficiency of the proposed framework, we present applications to some $L^{p}$-constrained least-squares problems.

An Approximate Penalty Method with Descent for Convex Optimization Problems

We propose a penalty method for general convex constrained optimization problems, where each auxiliary penalized problem is solved approximately with a special composite descent method. Direction finding choice in this method is found with the help of an equivalent equilibrium type problem. This allows one to keep the complete structure of the initial problem, although without nonlinear constraints and to simply calculate the descent direction in separable problems. Convergence of the method in primal and dual variables is established under rather weak assumptions.

Distributed Alternating Direction Method of Multipliers for Linearly Constrained Optimization Over a Network

In this letter we address the distributed optimization problem for a network of agents, which commonly occurs in several control engineering applications. Differently from the related literature, where only consensus constraints are typically addressed, we consider a challenging distributed optimization set-up where agents rely on local communication and computation to optimize a sum of local objective functions, each depending on individual variables subject to local constraints, while satisfying linear coupling constraints. Thanks to the distributed scheme, the resolution of the optimization problem turns into designing an iterative control procedure that steers the strategies of agents-whose dynamics is decoupled-not only to be convergent to the optimal value but also to satisfy the coupling constraints. Based on duality and consensus theory, we develop a proximal Jacobian alternating direction method of multipliers (ADMM) for solving such a kind of linearly constrained convex optimization problems over a network. Using the monotone operator and fixed point mapping, we analyze the optimality of the proposed algorithm and establish its o(1/t) convergence rate. Finally, through numerical simulations we show that the proposed algorithm offers higher computational performances than recent distributed ADMM variants.

Constrained Quaternion-Variable Convex Optimization: A Quaternion-Valued Recurrent Neural Network Approach.

This paper proposes a quaternion-valued one-layer recurrent neural network approach to resolve constrained convex function optimization problems with quaternion variables. Leveraging the novel generalized Hamilton-real (GHR) calculus, the quaternion gradient-based optimization techniques are proposed to derive the optimization algorithms in the quaternion field directly rather than the methods of decomposing the optimization problems into the complex domain or the real domain. Via chain rules and Lyapunov theorem, the rigorous analysis shows that the deliberately designed quaternion-valued one-layer recurrent neural network stabilizes the system dynamics while the states reach the feasible region in finite time and converges to the optimal solution of the considered constrained convex optimization problems finally. Numerical simulations verify the theoretical results.

Decentralized hierarchical constrained convex optimization

This paper proposes a decentralized optimization algorithm for the triple-hierarchical constrained convex optimization problem of minimizing a sum of strongly convex functions subject to a paramonotone variational inequality constraint over an intersection of fixed point sets of nonexpansive mappings. The existing algorithms for solving this problem are centralized optimization algorithms using all the information in the problem, and these algorithms are effective, but only under certain additional restrictions. The main contribution of this paper is to present a convergence analysis of the proposed algorithm in order to show that the proposed algorithm using incremental gradients with diminishing step-size sequences converges to the solution to the problem without any additional restrictions. Another contribution of this paper is the elucidation of the practical applications of hierarchical constrained optimization in the form of network resource allocation and optimal control problems. In particular, it is shown that the proposed algorithm can be applied to decentralized network resource allocation with a triple-hierarchical structure.