Block Coordinate Descent Approach Research Articles

Consistent adaptive sequential dictionary learning

Abstract Algorithms for learning overcomplete dictionaries for sparse signal representation are mostly iterative minimization methods that alternate between a sparse coding stage and a dictionary update stage. For most however, the notion of consistency of the learned quantities has not been addressed. Based on the observation that the observed signals can be approximated as a sum of rank one matrices, a new adaptive dictionary learning algorithm is proposed in this paper. It is derived via sequential adaptive penalized rank one matrix approximation where the l1-norm is introduced as a penalty promoting sparsity. The proposed algorithm uses a block coordinate descent approach to consistently estimate the unknowns and has the advantage of having simple closed form solutions for both the sparse coding and dictionary update stages. The consistency properties of both the estimated sparse code and dictionary atom are provided. The performance of the proposed algorithm compared to some state of the art algorithms is illustrated on both simulated data and a real functional magnetic resonance imaging (fMRI) data set from a finger tapping experiment.

Error-Correcting Factorization.

Error Correcting Output Codes (ECOC) is a successful technique in multi-class classification, which is a core problem in Pattern Recognition and Machine Learning. A major advantage of ECOC over other methods is that the multi-class problem is decoupled into a set of binary problems that are solved independently. However, literature defines a general error-correcting capability for ECOCs without analyzing how it distributes among classes, hindering a deeper analysis of pair-wise error-correction. To address these limitations this paper proposes an Error-Correcting Factorization (ECF) method. Our contribution is three fold: (I) We propose a novel representation of the error-correction capability, called the design matrix, that enables us to build an ECOC on the basis of allocating correction to pairs of classes. (II) We derive the optimal code length of an ECOC using rank properties of the design matrix. (III) ECF is formulated as a discrete optimization problem, and a relaxed solution is found using an efficient constrained block coordinate descent approach. (IV) Enabled by the flexibility introduced with the design matrix we propose to allocate the error-correction on classes that are prone to confusion. Experimental results in several databases show that when allocating the error-correction to confusable classes ECF outperforms state-of-the-art approaches.

Research on joint segment optimisation and stereo matching

Image segments are often used as a constraint in stereo matching. However, both over-segmentation and under-segmentation can lead to disparity degradation in some regions. To obtain an accurate disparity map, a modified semi-global matching (SGM) algorithm is proposed which is based on adaptive window models. Introducing the object notion and build a new global energy function to optimise segments and estimate a disparity map jointly. The effective and efficient block coordinate descent approach is used to optimise the global energy function by merging small segments. The authors’ demonstrate the performance of the proposed algorithm on the KITTI and Middlebury benchmarks. The results show that the authors’ algorithm outperforms many state-of-the-art methods and confirm the effectiveness of approach.

Efficient Sum of Outer Products Dictionary Learning (SOUP-DIL) and Its Application to Inverse Problems.

The sparsity of signals in a transform domain or dictionary has been exploited in applications such as compression, denoising and inverse problems. More recently, data-driven adaptation of synthesis dictionaries has shown promise compared to analytical dictionary models. However, dictionary learning problems are typically non-convex and NP-hard, and the usual alternating minimization approaches for these problems are often computationally expensive, with the computations dominated by the NP-hard synthesis sparse coding step. This paper exploits the ideas that drive algorithms such as K-SVD, and investigates in detail efficient methods for aggregate sparsity penalized dictionary learning by first approximating the data with a sum of sparse rank-one matrices (outer products) and then using a block coordinate descent approach to estimate the unknowns. The resulting block coordinate descent algorithms involve efficient closed-form solutions. Furthermore, we consider the problem of dictionary-blind image reconstruction, and propose novel and efficient algorithms for adaptive image reconstruction using block coordinate descent and sum of outer products methodologies. We provide a convergence study of the algorithms for dictionary learning and dictionary-blind image reconstruction. Our numerical experiments show the promising performance and speedups provided by the proposed methods over previous schemes in sparse data representation and compressed sensing-based image reconstruction.

A block coordinate descent approach for sparse principal component analysis

There are mainly two methodologies utilized in current sparse PCA calculation, the greedy approach and the block approach. While the greedy approach tends to be incrementally invalidated in sequentially generating sparse PCs due to the cumulation of computational errors, the block approach is difficult to elaborately rectify individual sparse PCs under certain practical sparsity or nonnegative constraints. In this paper, a simple while effective block coordinate descent (BCD) method is proposed for solving the sparse PCA problem. The main idea is to separate the original sparse PCA problem into a series of simple sub-problems, each having a closed-form solution. By cyclically solving these sub-problems in an analytical way, the BCD algorithm can be easily constructed. Despite its simplicity, the proposed method performs surprisingly well in extensive experiments implemented on a series of synthetic and real data. In specific, as compared to the greedy approach, the proposed method can iteratively ameliorate the deviation errors of all computed sparse PCs and avoid the problem of accumulating errors; as compared to the block approach, the proposed method can easily handle the constraints imposed on each individual sparse PC, such as certain sparsity and/or nonnegativity constraints. Besides, the proposed method converges to a stationary point of the problem, and its computational complexity is approximately linear in both data size and dimensionality, which makes it well suited to handle large-scale problems of sparse PCA.

Dynamic Spectrum Management With Spherical Coordinates

Multiuser interference, i.e., crosstalk, is the main bottleneck for digital subscriber lines (DSL) technology. Dynamic spectrum management (DSM) mitigates crosstalk by focusing on the multiuser power/frequency resource allocation problem, and it can provide formidable gains in performance. In this paper, we look at the DSM problem from a different perspective. We formulate the problem with the power allocation vectors defined with spherical coordinates, i.e., as a function of a radius and angles. We see that this reformulation permits us to exploit structure in the problem. We propose two algorithms. In the first of them, we use the fact that the DSM problem is concave in the radial dimension and perform an exhaustive search for the angles. The second algorithm uses a block coordinate descent approach, i.e., a sequence of line searches. We show that there is structure to be found in the radial dimension (it is always concave) and in the angle dimensions. For the latter, we provide conditions for the line searches to be concave or convex for each of the angles. The fact that we use structure leads to large savings in computational complexity. For example, we see that our first algorithm can be up to 60 times faster than a corresponding previously proposed algorithm. Our second algorithm is 2-15 times faster than a relevant previously proposed algorithm.

A Unified Convergence Analysis of Block Successive Minimization Methods for Nonsmooth Optimization

The block coordinate descent (BCD) method is widely used for minimizing a continuous function $f$ of several block variables. At each iteration of this method, a single block of variables is optimized, while the remaining variables are held fixed. To ensure the convergence of the BCD method, the subproblem of each block variable needs to be solved to its unique global optimal. Unfortunately, this requirement is often too restrictive for many practical scenarios. In this paper, we study an alternative inexact BCD approach which updates the variable blocks by successively minimizing a sequence of approximations of $f$ which are either locally tight upper bounds of $f$ or strictly convex local approximations of $f$. The main contributions of this work include the characterizations of the convergence conditions for a fairly wide class of such methods, especially for the cases where the objective functions are either nondifferentiable or nonconvex. Our results unify and extend the existing convergence results ...

Robust Clustering Using Outlier-Sparsity Regularization

Notwithstanding the popularity of conventional clustering algorithms such as K-means and probabilistic clustering, their clustering results are sensitive to the presence of outliers in the data. Even a few outliers can compromise the ability of these algorithms to identify meaningful hidden structures rendering their outcome unreliable. This paper develops robust clustering algorithms that not only aim to cluster the data, but also to identify the outliers. The novel approaches rely on the infrequent presence of outliers in the data, which translates to sparsity in a judiciously chosen domain. Leveraging sparsity in the outlier domain, outlier-aware robust K-means and probabilistic clustering approaches are proposed. Their novelty lies on identifying outliers while effecting sparsity in the outlier domain through carefully chosen regularization. A block coordinate descent approach is developed to obtain iterative algorithms with convergence guarantees and small excess computational complexity with respect to their non-robust counterparts. Kernelized versions of the robust clustering algorithms are also developed to efficiently handle high-dimensional data, identify nonlinearly separable clusters, or even cluster objects that are not represented by vectors. Numerical tests on both synthetic and real datasets validate the performance and applicability of the novel algorithms.

A heuristic block coordinate descent approach for controlled tabular adjustment

One of the main concerns of national statistical agencies (NSAs) is to publish tabular data. NSAs have to guarantee that no private information from specific respondents can be disclosed from the released tables. The purpose of the statistical disclosure control field is to avoid such a leak of private information. Most protection techniques for tabular data rely on the formulation of a large mathematical programming problem, whose solution is computationally expensive even for tables of moderate size. One of the emerging techniques in this field is controlled tabular adjustment (CTA). Although CTA is more efficient than other protection methods, the resulting mixed integer linear problems (MILP) are still challenging. In this work a heuristic approach based on block coordinate descent decomposition is designed and applied to large hierarchical and general CTA instances. This approach is compared with CPLEX, a state-of-the-art MILP solver. Our results, from both synthetic and real tables with up to 1,200,000 cells, 100,000 of them being sensitive (resulting in MILP instances of up to 2,400,000 continuous variables, 100,000 binary variables, and 475,000 constraints) show that the heuristic block coordinate descent has a better practical behavior than a state-of-the-art solver: for large hierarchical instances it provides significantly better solutions within a specified realistic time limit, as required by NSAs in real-world.

Distributed Estimation in Clustered Wireless Sensor Networks

We consider the problem of distributed estimation of a deterministic vector parameters using in clustered wireless sensor networks (WSNs). The paper using the method of multipliers in conjunction with a block coordinate descent approach, we demonstrate how the resultant algorithm can be decomposed into a set of simpler tasks suitable for distributed implementation based on maximum likelihood estimators (MLE) in nonlinear and non-Gaussian data models. the iterative algorithms based on the communication between nodes and the head node that generate a local estimation, and the head nodes can be seen as the neighbor node of all the other nodes in the cluster, it can broadcast its estimation in the cluster, and the local iterates converge to the global MLE, We prove that these algorithms have guaranteed convergence to the desired estimator when the sensor links are assumed ideal. Furthermore, corroborating simulations demonstrate the merits of the novel distributed estimation algorithms.

Fast Nonnegative Matrix Factorization: An Active-Set-Like Method and Comparisons

Nonnegative matrix factorization (NMF) is a dimension reduction method that has been widely used for numerous applications, including text mining, computer vision, pattern discovery, and bioinformatics. A mathematical formulation for NMF appears as a nonconvex optimization problem, and various types of algorithms have been devised to solve the problem. The alternating nonnegative least squares (ANLS) framework is a block coordinate descent approach for solving NMF, which was recently shown to be theoretically sound and empirically efficient. In this paper, we present a novel algorithm for NMF based on the ANLS framework. Our new algorithm builds upon the block principal pivoting method for the nonnegativity-constrained least squares problem that overcomes a limitation of the active set method. We introduce ideas that efficiently extend the block principal pivoting method within the context of NMF computation. Our algorithm inherits the convergence property of the ANLS framework and can easily be extended to other constrained NMF formulations. Extensive computational comparisons using data sets that are from real life applications as well as those artificially generated show that the proposed algorithm provides state-of-the-art performance in terms of computational speed.

Consensus in Ad Hoc WSNs With Noisy Links—Part I: Distributed Estimation of Deterministic Signals

We deal with distributed estimation of deterministic vector parameters using ad hoc wireless sensor networks (WSNs). We cast the decentralized estimation problem as the solution of multiple constrained convex optimization subproblems. Using the method of multipliers in conjunction with a block coordinate descent approach we demonstrate how the resultant algorithm can be decomposed into a set of simpler tasks suitable for distributed implementation. Different from existing alternatives, our approach does not require the centralized estimator to be expressible in a separable closed form in terms of averages, thus allowing for decentralized computation even of nonlinear estimators, including maximum likelihood estimators (MLE) in nonlinear and non-Gaussian data models. We prove that these algorithms have guaranteed convergence to the desired estimator when the sensor links are assumed ideal. Furthermore, our decentralized algorithms exhibit resilience in the presence of receiver and/or quantization noise. In particular, we introduce a decentralized scheme for least-squares and best linear unbiased estimation (BLUE) and establish its convergence in the presence of communication noise. Our algorithms also exhibit potential for higher convergence rate with respect to existing schemes. Corroborating simulations demonstrate the merits of the novel distributed estimation algorithms.