Inexact Alternating Direction Method Research Articles

Federated Learning via Inexact ADMM.

One of the crucial issues in federated learning is how to develop efficient optimization algorithms. Most of the current ones require full device participation and/or impose strong assumptions for convergence. Different from the widely-used gradient descent-based algorithms, in this paper, we develop an inexact alternating direction method of multipliers (ADMM), which is both computation- and communication-efficient, capable of combating the stragglers' effect, and convergent under mild conditions. Furthermore, it has high numerical performance compared with several state-of-the-art algorithms for federated learning.

A Multilevel Heterogeneous ADMM Algorithm for Elliptic Optimal Control Problems with L1-Control Cost

In this paper, elliptic optimal control problems with L1-control cost and box constraints on the control are considered. To numerically solve the optimal control problems, we use the First optimize, then discretize approach. We focus on the inexact alternating direction method of multipliers (iADMM) and employ the standard piecewise linear finite element approach to discretize the subproblems in each iteration. However, in general, solving the subproblems is expensive, especially when the discretization is at a fine level. Motivated by the efficiency of the multigrid method for solving large-scale problems, we combine the multigrid strategy with the iADMM algorithm. Instead of fixing the mesh size before the computation process, we propose the strategy of gradually refining the grid. Moreover, to overcome the difficulty whereby the L1-norm does not have a decoupled form, we apply nodal quadrature formulas to approximately discretize the L1-norm and L2-norm. Based on these strategies, an efficient multilevel heterogeneous ADMM (mhADMM) algorithm is proposed. The total error of the mhADMM consists of two parts: the discretization error resulting from the finite-element discretization and the iteration error resulting from solving the discretized subproblems. Both errors can be regarded as the error of inexactly solving infinite-dimensional subproblems. Thus, the mhADMM can be regarded as the iADMM in function space. Furthermore, theoretical results on the global convergence, as well as the iteration complexity results o(1/k) for the mhADMM, are given. Numerical results show the efficiency of the mhADMM algorithm.

Alternating direction method for the fixed point problem of set-valued mappings with second-order cone double constraints

<abstract><p>This paper focuses on solving a class of equilibrium problems, namely, the fixed point problem of set-valued mappings with second-order cone double constraints. Under certain conditions, the variational inequality form of the fixed point problem of set-valued mappings with second-order cone double constraints is obtained by using the generalized saddle point theory three times. The alternating direction method is used to solve the fixed point problem of set-valued mappings with second-order cone double constraints, and the global convergence of the algorithm is proved. Finally, numerical results of solving five examples with an inexact alternating direction method are given, and the feasibility and effectiveness of the algorithm are demonstrated by comparing with other algorithms.</p></abstract>

FedGiA: An Efficient Hybrid Algorithm for Federated Learning

Federated learning has shown its advances recently but is still facing many challenges, such as how algorithms save communication resources and reduce computational costs, and whether they converge. To address these critical issues, we propose a hybrid federated learning algorithm (FedGiA) that combines the gradient descent and the inexact alternating direction method of multipliers. The proposed algorithm is more communication- and computation-efficient than several state-of-the-art algorithms theoretically and numerically. Moreover, it also converges globally under mild conditions.

An inexact ADMM with proximal-indefinite term and larger stepsize

In this paper, an inexact Alternating Direction Method of Multipliers (ADMM) has been proposed for solving the two-block separable convex optimization problem subject to linear equality constraints. The first resulting subproblem is solved inexactly under relative error criterion, while another subproblem called regularization problem is solved inexactly by introducing an indefinite proximal term. Meanwhile, the dual variable is updated twice with relatively larger stepsizes since an indefinite proximal term is exploited. By reformulating the first-order optimality conditions of the involved subproblems as a monotone inclusion problem, the sublinear convergence rate of the proposed algorithm is established in the pointwise and ergodic sense. Numerical experiments on testing two sparse optimization problems from statistical learning and image restoration indicate that our inexact ADMM performs much better than several well-established algorithms.

Optimal Dispatch of a Virtual Storage Plant Using Inexact ADMM

A virtual storage plant (VSP) is identified as an effective approach to aggregating distributed storage devices participating in power network supports with similar capability as bulk storage systems. In this study, we develop a distributed control framework for cost-effective storage coordination in the distribution networks, in which the energy storage units are coordinated to contribute to a given power reference at the aggregated level while regulating the local network voltages in the presence of renewable generations. The salient features of the proposed VSP control roots from the successful employment of an inexact alternating direction method of multiplier (ADMM) algorithm, in which the primal updates have analytical solutions in closed form using proximal operators, which significantly reduces the computation efforts of individual storage agents, and renders fast storage dispatch. The proposed control is favorable for near real-time storage dispatch in an optimal manner, and its effectiveness is demonstrated using realistic distributed networks in the simulations.

Finding the global optimum of a class of quartic minimization problem

We consider a special nonconvex quartic minimization problem over a single spherical constraint, which includes the discretized energy functional minimization problem of non-rotating Bose-Einstein condensates (BECs) as one of the important applications. Such a problem is studied by exploiting its characterization as a nonlinear eigenvalue problem with eigenvector nonlinearity (NEPv). Firstly, we show that the NEPv has a unique nonnegative eigenvector, corresponding to the smallest nonlinear eigenvalue of NEPv, which is exactly the global minimizer to the optimization problem. Secondly, with these properties, we obtain that any algorithm converging to the nonnegative stationary point of this optimization problem finds its global optimum, such as the regularized Newton method. In particular, we obtain the convergence to the global optimum of the inexact alternating direction method of multipliers for this problem. Numerical experiments for applications in non-rotating BECs validate our theories.

A Barzilai and Borwein regularization feasible direction algorithm for convex nonlinear SOC programming with linear constraints

A Barzilai and Borwein regularization feasible direction algorithm is proposed for convex nonlinear second-order cone programming with linear constraints. In the algorithm, a regularization penalty term based on the Barzilai and Borwein parameters is added to the Frank–Wolfe linearization objective function of the direction generating subproblem. The Barzilai and Borwein subproblem is transformed into a multi-block separable convex quadratic second-order cone programming (CQSOCP) subproblem. A parallel inexact alternating direction method is applied to solve the multi-block separable CQSOCP subproblem. The global convergence is given. Numerical results demonstrate that our method is efficient for some random convex nonlinear second-order cone programming problems with low accuracy.

Key Nodes Selection in Controlling Complex Networks via Convex Optimization.

Key nodes are the nodes connected with a given number of external source controllers that result in minimal control cost. Finding such a subset of nodes is a challenging task since it impossible to list and evaluate all possible solutions unless the network is small. In this paper, we approximately solve this problem by proposing three algorithms step by step. By relaxing the Boolean constraints in the original optimization model, a convex problem is obtained. Then inexact alternating direction method of multipliers (IADMMs) is proposed and convergence property is theoretically established. Based on the degree distribution, an extension method named degree-based IADMM (D-IADMM) is proposed such that key nodes are pinpointed. In addition, with the technique of local optimization employed on the results of D-IADMM, we also develop LD-IADMM and the performance is greatly improved. The effectiveness of the proposed algorithms is validated on different networks ranging from Erdős-Rényi networks and scale-free networks to some real-life networks.

Learnable MIMO Detection Networks Based on Inexact ADMM

In this article, we present a new iterative MIMO detection algorithm based on inexact alternating direction method of multipliers. Each iteration is considered as a neural network layer with learnable parameters, which are optimized by the stochastic gradient descent algorithm with a training data set of the received vectors and the ground truth transmitted signals. Numerical results show that the proposed algorithm outperforms the existing learnable detection network and it achieves near-optimal performance close to the sphere decoder in the case of a large number of receive antennas.

An inexact alternating direction method of multipliers for a kind of nonlinear complementarity problems

<p style='text-indent:20px;'>Many kinds of practical problems can be formulated as nonlinear complementarity problems. In this paper, an inexact alternating direction method of multipliers for the solution of a kind of nonlinear complementarity problems is proposed. The convergence analysis of the method is given. Numerical results confirm the theoretical analysis, and show that the proposed method can be more efficient and faster than the modulus-based Jacobi, Gauss-Seidel and Successive Overrelaxation method when the dimension of the problem being solved is large.

A prediction-correction inexact alternating direction method for convex nonlinear second-order cone programming with linear constraints

The convex nonlinear second-order cone programming with linear constraints is equivalent to a separate structure convex programming. A prediction-correction inexact alternating direction method is proposed for the separate structure convex programming. In the proposed method, the convex objective function is not required to be Lipschitz continuous and only needs satisfy an inequality. The global convergence result is given. Numerical results demonstrate that our method is efficient for some random second-order cone programming problems in lower accuracy. In addition, our method can be extended to the convex nonlinear circular cone programming with linear constraints. We also give the simulation results of the three-fingered grasping force optimization problems.

Image deconvolution for confocal laser scanning microscopy using constrained total variation with a gradient field

As for the confocal laser scanning microscope (CLSM) imaging system, the collected weak fluorescence signals are always distorted by optic blur and severe photon-counting noise, and the deconvolution for CLSM images is a typical ill-posed inverse problem, which is highly sensitive to the measurement noise. To promote the reconstruction quality for characteristics of low intensity and strong noise, we employed the prominent total variation regularization (TV) to enforce the sparsity of a fluorescent image gradient with rich details. However, the well-known reconstruction artifacts (e.g., artificial staircase) emerge with TV prior. To settle this issue, we utilized a robust first-order discretization yielding near-isotropy with a gradient field to depress the reconstruction artifacts. Furthermore, the bound constraint was suited to restrain final reconstruction results from appearing unreasonably explosive. For the proposed optimization minimizer with linear constraint, we take one proximal gradient for approximate estimation of each subproblem under the framework of the inexact alternating direction method of multipliers. Moreover, we incorporated a Nesterov's scheme into the numerical method for acceleration of iteration updating. Compared with other competing methods, both the simulation and practical results demonstrate the effectiveness of our proposed model for CLSM image deconvolution.

A Partially Inexact Proximal Alternating Direction Method of Multipliers and Its Iteration-Complexity Analysis

This paper proposes a partially inexact proximal alternating direction method of multipliers for computing approximate solutions of a linearly constrained convex optimization problem. This method allows its first subproblem to be solved inexactly using a relative approximate criterion, whereas a proximal term is added to its second subproblem in order to simplify it. A stepsize parameter is included in the updating rule of the Lagrangian multiplier to improve its computational performance. Pointwise and ergodic iteration-complexity bounds for the proposed method are established. To the best of our knowledge, this is the first time that complexity results for an inexact alternating direction method of multipliers with relative error criteria have been analyzed. Some preliminary numerical experiments are reported to illustrate the advantages of the new method.

Query-dependent ranking and its asymptotic properties

Ranking, also known as learning to rank in machine learning community, is to rank a number of items based on their relevance to a specific query. In literature, most ranking methods use a uniform ranking function to evaluate the relevance, which completely ignores the heterogeneity among queries. To admit different ranking functions for various queries, a general $U$-process formulation for query-dependent ranking is developed. It allows to incorporate neighborhood structure among queries via various forms of smoothing weights to improve the ranking performance. One of its salient features is its capability of producing reasonable rankings for novel queries that are absent in the training set, which is commonly encountered in practice but often neglected in the literature. The proposed method is implemented via an inexact alternating direction method of multipliers (ADMM) for each query parallelly. Its asymptotic risk bound is established, showing that it achieves desirable ranking accuracy at a fast rate for any query including the novel ones. Furthermore, simulated examples and a real application to the Yahoo! challenge dataset also support the advantage of the query-dependent ranking method against existing competitors.

Distributed Jointly Sparse Bayesian Learning With Quantized Communication

We consider the problem of the distributed recovery of jointly sparse signals by a number of nodes in a sensor network, using multiple noisy linear measurements. In the literature, distributed Bayesian algorithms have been proposed to tackle this problem, most of which assume that nodes can transmit real data with infinite precision. However, in many practical applications, sensor networks have a limited communication bandwidth and finite capacity channels, and digital quantization of the transmitted data is inevitable. In this paper, we consider the case that the transmitted quantities/messages between nodes (instead of the measurement data) are quantized with discrete value and finite precision. We formulate a fully hierarchical jointly sparse Bayesian model and propose a novel distributed variational Bayesian (VB) algorithm, which uses only the quantized transmitted messages. In the proposed VB algorithm, an inexact alternating direction method of multipliers is developed for achieving quantized consensus. We theoretically analyze the convergence of the proposed algorithm and study the effect of digital quantization. Through numerical simulations, we find a counterintuitive result that the proposed quantized algorithm can even perform better than the corresponding unquantized algorithm and the centralized counterpart.

An Efficient and Fast Quantum State Estimator With Sparse Disturbance.

A pure or nearly pure quantum state can be described as a low-rank density matrix, which is a positive semidefinite and unit-trace Hermitian. We consider the problem of recovering such a low-rank density matrix contaminated by sparse components, from a small set of linear measurements. This quantum state estimation task can be formulated as a robust principal component analysis (RPCA) problem subject to positive semidefinite and unit-trace Hermitian constraints. We propose an efficient and fast inexact alternating direction method of multipliers (I-ADMM), in which the subproblems are solved inexactly and hence have closed-form solutions. We prove global convergence of the proposed I-ADMM, and the theoretical result provides a guideline for parameter setting. Numerical experiments show that the proposed I-ADMM can recover state density matrices of 5 qubits on a laptop in 0.69 s, with 6 ×10-4 accuracy (99.38% fidelity) using 30% compressive sensing measurements, which outperforms existing algorithms.

A two‐phase strategy for control constrained elliptic optimal control problems

SummaryElliptic optimal control problems with pointwise box constraints on the control are considered. To numerically solve elliptic optimal control problems with pointwise box constraints on the control, an inexact alternating direction method of multipliers (iADMM) is first proposed on the continuous level with the aim of solving discretized problems with moderate accuracy. Then, the standard piecewise linear finite element is employed to discretize the related subproblems appearing in each iteration of the iADMM algorithm. Such approach will give us the freedom to discretize two inner subproblems of the iADMM algorithm by different discretized scheme, respectively. More importantly, it should be emphasized that the discretized version of the iADMM algorithm can be regarded as a modification of the inexact semiproximal ADMM (isPADMM) algorithm. In order to obtain more accurate solution, the primal‐dual active set method is used as a postprocessor of the isPADMM. Numerical results not only show that the isPADMM and the two‐phase strategy are highly efficient but also show the mesh independence of the isPADMM.

An inexact alternating direction method of multipliers for the solution of linear complementarity problems arising from free boundary problems

A large number of free boundary problems can be formulated as linear-complementarity problems. In this paper, we propose an inexact alternating direction method of multipliers for solving linear complementarity problem arising from free boundary problems by using the special structure of these problems. The convergence of our proposed method is proved. Numerical results show that the proposed method is feasible and effective, and it is significantly faster than modified alternating direction implicit algorithm and many other methods, especially when dimension of the problem being solved is large.

Distributed Learning of Predictive Structures From Multiple Tasks Over Networks

This paper is concerned with the problem of distributed multitask learning over networks, which aims to simultaneously infer multiple node-specific parameter vectors in a collaborative manner. Most of the existing works on the distributed multitask problem modeled the task relatedness by assuming some similarities of parameter vectors in an explicit way. In this paper, we implicitly model the similarity of parameter vectors by assuming that the parameter vectors share a common low-dimensional predictive structure on hypothesis spaces, which is learned using the available data in networks. A distributed structure learning algorithm for the in-network cooperative estimation problem is then derived based on the block coordinate descent method integrated with the inexact alternating direction method of multipliers technique. Simulations on both synthetic and real-world datasets are given to verify the effectiveness of the proposed algorithm. In the case that each node shares a common predictive subspace, it is demonstrated that the proposed multitask algorithm outperforms the noncooperative learning algorithm. Moreover, the use of the inexact approach can significantly reduce the communication bandwidth and still provide the same optimal solution as the corresponding centralized approach.