Subgradient Scheme Research Articles

Iterative Newton hard thresholding‐based approach for cosparse recovery of mmWave massive MIMO channels

SummaryMillimeter wave (mmWave) massive multiple‐input multiple‐output (MIMO) channels are cosparse in nature. These cosparse channels share common cosparsity properties, which is interesting when it comes to estimating a large number of mmWave mMIMO channels. Therefore, the channel estimation (CE) problem can be solved by exploiting the cosparsity and common cosupport properties. Specifically, the CE problem can be formulated as a cosparse signal recovery problem, which can be solved using an analysis operator (AO) and the iterative hard thresholding (IHT) algorithm. Prior analysis‐based compressive sensing (ACS) CE techniques adopt an overcomplete AO, which leads usually to a suboptimal solution that cannot guarantee CE accuracy. In this paper, we firstly propose a projected subgradient (PSG) scheme for AO learning. Then, using the previously learned AO, we propose an iterative Newton hard thresholding (INHT)‐based ACS‐CE algorithm. The effectiveness of the proposal has been proven by computational complexity analysis and numerical simulations.

Distributionally Robust Optimization with Confidence Bands for Probability Density Functions

Distributionally robust optimization (DRO) has been introduced for solving stochastic programs in which the distribution of the random variables is unknown and must be estimated by samples from that distribution. A key element of DRO is the construction of the ambiguity set, which is a set of distributions that contains the true distribution with a high probability. Assuming that the true distribution has a probability density function, we propose a class of ambiguity sets based on confidence bands of the true density function. As examples, we consider the shape-restricted confidence bands and the confidence bands constructed with a kernel density estimation technique. The former allows us to incorporate the prior knowledge of the shape of the underlying density function (e.g., unimodality and monotonicity), and the latter enables us to handle multidimensional cases. Furthermore, we establish the convergence of the optimal value of DRO to that of the underlying stochastic program as the sample size increases. The DRO with our ambiguity set involves functional decision variables and infinitely many constraints. To address this challenge, we apply duality theory to reformulate the DRO to a finite-dimensional stochastic program, which is amenable to a stochastic subgradient scheme as a solution method.

On the resolution of misspecified convex optimization and monotone variational inequality problems

We consider a misspecified optimization problem, requiring the minimization of a function $$f(\cdot;\theta ^*)$$ over a closed and convex set X where $$\theta ^*$$ is an unknown vector of parameters that may be learnt by a parallel learning process. Here, we develop coupled schemes that generate iterates $$(x_k,\theta _k)$$ as $$k \rightarrow \infty$$ , then $$x_k \rightarrow x^*$$ , a minimizer of $$f(\cdot;\theta ^*)$$ over X and $$\theta _k \rightarrow \theta ^*$$ . In the first part of the paper, we consider the solution of problems where f is either smooth or nonsmooth. In smooth strongly convex regimes, we demonstrate that such schemes still display a linear rate of convergence, albeit with larger constants. When strong convexity assumptions are weakened, it can be shown that the convergence in function value sees a modification in the canonical convergence rate of $${{{\mathcal {O}}}}(1/K)$$ by an additive factor proportional to $$\Vert \theta _0-\theta ^*\Vert$$ where $$\Vert \theta _0-\theta ^*\Vert$$ represents the initial misspecification in $$\theta ^*$$ . In addition, when the learning problem is assumed to be merely convex but admits a suitable weak-sharpness property, then the convergence rate deteriorates to $${\mathcal {O}}(1/\sqrt{K})$$ . In both convex and strongly convex regimes, diminishing steplength schemes are also provided and are less reliant on the knowledge of problem parameters. Finally, we present an averaging-based subgradient scheme that displays a rate of $${\mathcal {O}}(1/\sqrt{K})+ \mathcal{O}(\|\theta_0-\theta^*\|(1/K))$$ , implying no effect on the canonical rate of $${{{\mathcal {O}}}}(1/\sqrt{K})$$ . In the second part of the paper, we consider the solution of misspecified monotone variational inequality problems, motivated by the need to contend with more general equilibrium problems as well as the possibility of misspecification in the constraints. In this context, we first present a constant steplength misspecified extragradient scheme and prove its asymptotic convergence. This scheme is reliant on problem parameters (such as Lipschitz constants) and leads to a misspecified variant of iterative Tikhonov regularization, an avenue that does not necessitate the knowledge of such constants.

Robust stochastic optimization with convex risk measures: A discretized subgradient scheme

<p style='text-indent:20px;'>We study the distributionally robust stochastic optimization problem within a general framework of risk measures, in which the ambiguity set is described by a spectrum of practically used probability distribution constraints such as bounds on mean-deviation and entropic value-at-risk. We show that a subgradient of the objective function can be obtained by solving a finite-dimensional optimization problem, which facilitates subgradient-type algorithms for solving the robust stochastic optimization problem. We develop an algorithm for two-stage robust stochastic programming with conditional value at risk measure. A numerical example is presented to show the effectiveness of the proposed method.

Random Minibatch Subgradient Algorithms for Convex Problems with Functional Constraints

In this paper we consider non-smooth convex optimization problems with (possibly) infinite intersection of constraints. In contrast to the classical approach, where the constraints are usually represented as intersection of simple sets, which are easy to project onto, in this paper we consider that each constraint set is given as the level set of a convex but not necessarily differentiable function. For these settings we propose subgradient iterative algorithms with random minibatch feasibility updates. At each iteration, our algorithms take a subgradient step aimed at only minimizing the objective function and then a subsequent subgradient step minimizing the feasibility violation of the observed minibatch of constraints. The feasibility updates are performed based on either parallel or sequential random observations of several constraint components. We analyze the convergence behavior of the proposed algorithms for the case when the objective function is strongly convex and with bounded subgradients, while the functional constraints are endowed with a bounded first-order black-box oracle. For a diminishing stepsize, we prove sublinear convergence rates for the expected distances of the weighted averages of the iterates from the constraint set, as well as for the expected suboptimality of the function values along the weighted averages. Our convergence rates are known to be optimal for subgradient methods on this class of problems. Moreover, the rates depend explicitly on the minibatch size and show when minibatching helps a subgradient scheme with random feasibility updates.

Distributed multi-step subgradient optimization for multi-agent system

This paper proposes a novel multi-step subgradient scheme to solve distributed optimization problem with a balanced digraph. To this end, matrix diameter and radius are first introduced to describe the convergence rate of the adjacency matrix. Then, we propose a hierarchical and recursive observer to estimate the desired optimal solution, and we prove that the proposed strategy can asymptotically obtain an optimal solution, while all networked systems achieve a consensus among estimates in a finite-time. Simulation results verified the effectiveness of the proposed algorithm.

Latent Variable Graphical Model Selection using Harmonic Analysis: Applications to the Human Connectome Project (HCP).

A major goal of imaging studies such as the (ongoing) Human Connectome Project (HCP) is to characterize the structural network map of the human brain and identify its associations with covariates such as genotype, risk factors, and so on that correspond to an individual. But the set of image derived measures and the set of covariates are both large, so we must first estimate a 'parsimonious' set of relations between the measurements. For instance, a Gaussian graphical model will show conditional independences between the random variables, which can then be used to setup specific downstream analyses. But most such data involve a large list of 'latent' variables that remain unobserved, yet affect the 'observed' variables sustantially. Accounting for such latent variables is not directly addressed by standard precision matrix estimation, and is tackled via highly specialized optimization methods. This paper offers a unique harmonic analysis view of this problem. By casting the estimation of the precision matrix in terms of a composition of low-frequency latent variables and high-frequency sparse terms, we show how the problem can be formulated using a new wavelet-type expansion in non-Euclidean spaces. Our formulation poses the estimation problem in the frequency space and shows how it can be solved by a simple sub-gradient scheme. We provide a set of scientific results on ~500 scans from the recently released HCP data where our algorithm recovers highly interpretable and sparse conditional dependencies between brain connectivity pathways and well-known covariates.

A Piecewise Monotone Subgradient Algorithm for Accurate &lt;formula formulatype="inline"&gt;&lt;tex Notation="TeX"&gt;${\rm L}^{1}$&lt;/tex&gt;&lt;/formula&gt;-TV Based Registration of Physical Slices With Discontinuities in Microscopy

Image registration tasks are often formulated in terms of minimization of a functional consisting of a data fidelity term penalizing the mismatch between the reference and the target image, and a term enforcing smoothness of shift between neighboring pairs of pixels (a min-sum problem). Most methods for deformable image registration use some form of interpolation between matching control points. The interpolation makes it impossible to account for isolated discontinuities in the deformation field that may appear, e.g., when a physical slice of a microscopy specimen is ruptured by the cutting tool. For registration of neighboring physical slices of microscopy specimens with discontinuities, Janácek proposed an L¹-distance data fidelity term and a total variation (TV) smoothness term, and used a graph-cut (GC) based iterative steepest descent algorithm for minimization. The L¹-TV functional is nonconvex; hence a steepest descent algorithm is not guaranteed to converge to the global minimum. Schlesinger presented transformation of max-sum problems to minimization of a dual quantity called problem power, which is--contrary to the original max-sum functional--convex. Based on Schlesinger's solution to max-sum problems we developed an algorithm for L¹-TV minimization by iterative multi-label steepest descent minimization of the convex dual problem. For Schlesinger's subgradient algorithm we proposed a novel step control heuristics that considerably enhances both speed and accuracy compared with standard step size strategies for subgradient methods. It is shown experimentally that our subgradient scheme achieves consistently better image registration than GC in terms of lower values both of the composite L¹-TV functional, and of its components, i.e., the L¹ distance of the images and the transformation smoothness TV, and yields visually acceptable results even in cases where the GC based algorithm fails. The new algorithm allows easy parallelization and can thus be sped up by running on multi-core graphic processing units.

MRF Energy Minimization and Beyond via Dual Decomposition

This paper introduces a new rigorous theoretical framework to address discrete MRF-based optimization in computer vision. Such a framework exploits the powerful technique of Dual Decomposition. It is based on a projected subgradient scheme that attempts to solve an MRF optimization problem by first decomposing it into a set of appropriately chosen subproblems, and then combining their solutions in a principled way. In order to determine the limits of this method, we analyze the conditions that these subproblems have to satisfy and demonstrate the extreme generality and flexibility of such an approach. We thus show that by appropriately choosing what subproblems to use, one can design novel and very powerful MRF optimization algorithms. For instance, in this manner we are able to derive algorithms that: 1) generalize and extend state-of-the-art message-passing methods, 2) optimize very tight LP-relaxations to MRF optimization, and 3) take full advantage of the special structure that may exist in particular MRFs, allowing the use of efficient inference techniques such as, e.g., graph-cut-based methods. Theoretical analysis on the bounds related with the different algorithms derived from our framework and experimental results/comparisons using synthetic and real data for a variety of tasks in computer vision demonstrate the extreme potentials of our approach.

A tabu search heuristic procedure for the capacitated facility location problem

A tabu search heuristic procedure is developed, implemented and computationally tested for the capacitated facility location problem. The procedure uses different memory structures. Visited solutions are stored in a primogenitary linked quad tree. For each facility, the recent move at which the facility changed its status and the frequency it has been open are also stored. These memory structures are used to guide the main search process as well as the diversification and intensification processes. Lower bounds on the decreases of total cost are used to measure the attractiveness of the moves and to select moves in the search process. A specialized network algorithm is developed to exploit the problem structure in solving transportation problems. Criterion altering, solution reconciling and path relinking are used to perform intensification functions. The performance of the procedure is tested through computational experiments using test problems from the literature and new test problems randomly generated. It found optimal solutions for almost all test problems from the literature. As compared to the heuristic method of Lagrangean relaxation with improved subgradient scheme, the tabu search heuristic procedure found much better solutions using much less CPU time.

Lagrangian heuristic for a class of the generalized assignment problems

A Lagrangian based heuristic is proposed for many-to-many assignment problems taking into account capacity limits for task and agents. A modified Lagrangian bound studied earlier by the authors is presented and a greedy heuristic is then applied to get a feasible Lagrangian-based solution. The latter is also used to speed up the subgradient scheme to solve the modified Lagrangian dual problem. A numerical study is presented to demonstrate the efficiency of the proposed approach.

An Effective Deflected Subgradient Optimization Scheme for Implementing Column Generation for Large-Scale Airline Crew Scheduling Problems

We present a new deflected subgradient scheme for generating good quality dual solutions for linear programming (LP) problems and utilize this within the context of large-scale airline crew planning problems that arise in practice. The motivation for the development of this method came from the failure of a black-box-type approach implemented at United Airlines for solving such problems using column generation in concert with a commercial LP solver, where the software was observed to stall while yet remote from optimality. We identify a phenomenon called dual noise to explain this stalling behavior and present an analysis of the desirable properties of dual solutions in this context. The proposed deflected subgradient approach has been embedded within the crew pairing solver at United Airlines and tested using historical data sets. Our computational experience suggests a strong correlation between the dual noise phenomenon and the quality of the final solution produced, as well as with the accompanying algorithmic performance. Although we observed that our deflected subgradient scheme yielded an average speed-up factor of 10 for the column generation scheme over the commercial solver, the average reduction in the optimality gap over the same number of iterations was better by a factor of 26, along with an average reduction in the dual noise by a factor of 30. The results from the column generation implementation suggest that significant benefits can be obtained by using the deflected subgradient-based scheme instead of a black-box-type or standard solver approach to solve the intermediate linear programs that arise within the column generation scheme.

Ergodic, primal convergence in dual subgradient schemes for convex programming

Lagrangean dualization and subgradient optimization techniques are frequently used within the field of computational optimization for finding approximate solutions to large, structured optimization problems. The dual subgradient scheme does not automatically produce primal feasible solutions; there is an abundance of techniques for computing such solutions (via penalty functions, tangential approximation schemes, or the solution of auxiliary primal programs), all of which require a fair amount of computational effort. We consider a subgradient optimization scheme applied to a Lagrangean dual formulation of a convex program, and construct, at minor cost, an ergodic sequence of subproblem solutions which converges to the primal solution set. Numerical experiments performed on a traffic equilibrium assignment problem under road pricing show that the computation of the ergodic sequence results in a considerable improvement in the quality of the primal solutions obtained, compared to those generated in the basic subgradient scheme.

A Lagrangean Relaxation Scheme for Structured Linear Programs With Application To Multicommodity Network Flows

It is well known that linear programs can generally not be solved by straightforward Lagrangean relaxation and dual subgradient optimization, the reason being that the solutions to the Lagrangean relaxed problems are, normally, infeasible in the original linear program. This property is a consequence of the linearity of the problem and it holds even in the unlikely case that an exact optimal dual solution is found. We show that an optimal solution to the linear program can be obtained by calculating the simple average of the solutions to the relaxed problems which are solved during the subgradient search scheme, provided that the steplengths in this scheme are chosen according to a modified harmonic series. This method is similar to a procedure given earlier by Shor, which utilizes a particular weighted average and holds for any divergent series steplength rule. As an application of these two averaging schemes, we construct an approximate algorithm for the minimum cost multicommodity network flow problem. The algorithm has been implemented for solving multicommodity transportation problems and shows a good performance. The computational results indicate that it may be possible to use Lagrangean relaxation and dual subgradient optimization for constructing efficient special–purpose solution methods for structured large–scale linear programs

A dual scheme for traffic assignment problems

A solution method based on Lagrangean dualization and subgradient optimization for the symmetric traffic equilibrium assignment problem is presented. Its interesting feature is that it includes a simple and computationally cheap procedure for calculating a sequence of feasible flow assignments which tend to equilibrium ones. The Lagrangean subproblem essentially consists of shortest route searches, and it is shown that one may compute an equilibrium flow by taking the simple average of all the shortest route flows obtained during the subgradient optimization scheme, provided that its step lengths are chosen according to a modified harmonic series. The new method is compared to the Frank-Wolfe algorithm and the method of successive averages on a medium-scale problem; its computational performance is at least comparable to that of the two other methods. The main motive for considering this computational methodology is that it may easily be extended and applied to more complex traffic problems; this feature is shared by neither the Frank-Wolfe method nor the state-of-the-art methods for traffic assignment. We outline its extensions to several well-known network models, and illustrate the methodology numerically on one of these, an equilibrium model with link counts;the results obtained are encouraging