Coordinate Ascent Algorithm Research Articles

Topic-aware joint analysis of overlapping communities and roles in social media

Topic modeling can be used to improve the mutuality and interpenetration of community discovery and role analysis in social media. Also, it is useful to uncover communities and roles that are both social and topic-aware. In the present manuscript, we explore the exploitation of topic modeling to inform the seamless integration of community discovery and role analysis. For this purpose, we develop an innovative generative model of social media, in which the interrelation among communities, roles and topics is explained from a fully Bayesian perspective. Essentially, communities, roles and topics are latent factors that interact in an underlying generative process, to govern link formation and message wording. Posterior inference under the devised model allows for a variety of exploratory, descriptive and predictive tasks. These include the detection and interpretation of overlapping communities, roles and topics as well as the prediction of missing links. We derive the mathematical details of variational inference and design a coordinate-ascent algorithm implementing the latter. An empirical assessment on real-world social media demonstrates a superior accuracy of the proposed model in community discovery and link prediction compared to several established competitors, which substantiates the rationality of both our modeling effort and the underlying intuition.

Interference Mitigation via Relaying

This paper studies the effectiveness of relaying for interference mitigation in an interference-limited communication scenario. We are motivated by the observation that in a cellular network, a relay node placed at the cell edge observes a combination of intended signal and inter-cell interference that is correlated with the received signal at a nearby destination, so a relaying link can effectively allow the antennas at the relay and at the destination to be pooled together for both signal enhancement and interference mitigation. We model this scenario by a MIMO Gaussian relay channel with a digital relay-to-destination link of finite capacity, and with correlated noise across the relay and destination antennas. Assuming a compress-and-forward strategy with Gaussian input distribution and quantization noise, we propose a coordinate ascent algorithm for obtaining a stationary point of the non-convex joint optimization of the transmit and quantization covariance matrices. For fixed input distribution, the globally optimum quantization noise covariance matrix can be found in closed-form using a transformation of the relay's observation that simultaneously diagonalizes two conditional covariance matrices by congruence. For fixed quantization, the globally optimum transmit covariance matrix can be found via convex optimization. This paper further shows that such an optimized achievable rate is within a constant additive gap of the MIMO relay channel capacity. The optimal structure of the quantization noise covariance enables a characterization of the slope of the achievable rate as a function of the relay-to-destination link capacity. Moreover, this paper shows that the improvement in spatial degrees of freedom by MIMO relaying in the presence of noise correlation is related to the aforementioned slope via a connection to the deterministic relay channel.

Multivariate Skew Normal Copula for Asymmetric Dependence: Estimation and Application

The exchangeability and radial symmetry assumptions on the dependence structure of the multivariate data are restrictive in practical situations where the variables of interest are not likely to be associated to each other in an identical manner. In this paper, we propose a flexible class of multivariate skew normal copulas to model high-dimensional asymmetric dependence patterns. The proposed copulas have two sets of parameters capturing asymmetric dependence, one for association between the variables and the other for skewness of the variables. In order to efficiently estimate the two sets of parameters, we introduce the block coordinate ascent algorithm and discuss its convergence property. The proposed class of multivariate skew normal copulas is illustrated using a real data set.

Probabilistic Disjoint Principal Component Analysis

One of the most relevant problems in principal component analysis and factor analysis is the interpretation of the components/factors. In this paper, disjoint principal component analysis model is extended in a maximum-likelihood framework to allow for inference on the model parameters. A coordinate ascent algorithm is proposed to estimate the model parameters. The performance of the methodology is evaluated on simulated and real data sets.

SDP-based branch-and-bound for non-convex quadratic integer optimization

Semidefinite programming (SDP) relaxations have been intensively used for solving discrete quadratic optimization problems, in particular in the binary case. For the general non-convex integer case with box constraints, the branch-and-bound algorithm Q-MIST has been proposed by Buchheim and Wiegele (Math Program 141(1--2):435--452, 2013), which is based on an extension of the well-known SDP-relaxation for max-cut. For solving the resulting SDPs, Q-MIST uses an off-the-shelf interior point algorithm. In this paper, we present a tailored coordinate ascent algorithm for solving the dual problems of these SDPs. Building on related ideas of Dong (SIAM J Optim 26(3):1962--1985, 2016), it exploits the particular structure of the SDPs, most importantly a small rank of the constraint matrices. The latter allows both an exact line search and a fast incremental update of the inverse matrices involved, so that the entire algorithm can be implemented to run in quadratic time per iteration. Moreover, we describe how to extend this approach to a certain two-dimensional coordinate update. Finally, we explain how to include arbitrary linear constraints into this framework, and evaluate our algorithm experimentally.

Axially Symmetric Data Clustering Through Dirichlet Process Mixture Models of Watson Distributions.

This paper proposes a Bayesian nonparametric framework for clustering axially symmetric data. Our approach is based on a Dirichlet processes mixture model with Watson distributions, which can also be considered as the infinite Watson mixture model. In this paper, first, we extend the finite Watson mixture model into its infinite counterpart based on the framework of truncated Dirichlet process mixture model with a stick-breaking representation. Second, we propose a coordinate ascent mean-field variational inference algorithm that can effectively learn the parameters of our model with closed-form solutions; Third, to cope with a massive data set, we develop a stochastic variational inference algorithm to learn the proposed model through the method of stochastic gradient ascent; Finally, the proposed nonparametric Bayesian model is evaluated through simulated axially symmetric data sets and a real-world application, namely, gene expression data clustering.

Dual Free Adaptive Minibatch SDCA for Empirical Risk Minimization

In this paper, we develop an adaptive dual free Stochastic Dual Coordinate Ascent (adfSDCA) algorithm for regularized empirical risk minimization problems. This is motivated by the recent work on dual free SDCA of Shalev-Shwartz (2016). The novelty of our approach is that the coordinates to update at each iteration are selected non-uniformly from an adaptive probability distribution, and this extends the previously mentioned work which only allowed for a uniform selection of ``dual coordinates from a fixed probability distribution. We describe an efficient iterative procedure for generating the non-uniform samples, where the scheme selects the coordinate with the greatest potential to decrease the sub-optimality of the current iterate. We also propose a heuristic variant of adfSDCA that is more aggressive than the standard approach. Furthermore, in order to utilize multi-core machines, we consider a mini-batch adfSDCA algorithm and develop complexity results that guarantee the algorithm's convergence. The work is concluded with several numerical experiments to demonstrate the practical benefits of the proposed approach.

Pinball loss minimization for one-bit compressive sensing: Convex models and algorithms

Abstract The one-bit quantization is implemented by one single comparator that operates at low power and a high rate. Hence one-bit compressive sensing (1bit-CS) becomes attractive in signal processing. When measurements are corrupted by noise during signal acquisition and transmission, 1bit-CS is usually modeled as minimizing a loss function with a sparsity constraint. The one-sided l1 loss and the linear loss are two popular loss functions for 1bit-CS. To improve the decoding performance on noisy data, we consider the pinball loss, which provides a bridge between the one-sided l1 loss and the linear loss. Using the pinball loss, two convex models, an elastic-net pinball model and its modification with the l1-norm constraint, are proposed. To efficiently solve them, the corresponding dual coordinate ascent algorithms are designed and their convergence is proved. The numerical experiments confirm the effectiveness of the proposed algorithms and the performance of the pinball loss minimization for 1bit-CS.

Automatic Parameter Tying: A New Approach for Regularized Parameter Learning in Markov Networks

Parameter tying is a regularization method in which parameters (weights) of a machine learning model are partitioned into groups by leveraging prior knowledge and all parameters in each group are constrained to take the same value. In this paper, we consider the problem of parameter learning in Markov networks and propose a novel approach called automatic parameter tying (APT) that uses automatic instead of a priori and soft instead of hard parameter tying as a regularization method to alleviate overfitting. The key idea behind APT is to set up the learning problem as the task of finding parameters and groupings of parameters such that the likelihood plus a regularization term is maximized. The regularization term penalizes models where parameter values deviate from their group mean parameter value. We propose and use a block coordinate ascent algorithm to solve the optimization task. We analyze the sample complexity of our new learning algorithm and show that it yields optimal parameters with high probability when the groups are well separated. Experimentally, we show that our method improves upon L2 regularization and suggest several pragmatic techniques for good practical performance.

Acceleration for proximal stochastic dual coordinate ascent algorithm in solving regularised loss minimisation with norm

An accelerated version of the proximal stochastic dual coordinate ascent (SDCA) algorithm in solving regularised loss minimisation with norm is presented, wherein a momentum is introduced and the strong theoretical guarantees of SDCA are shared. Moreover, it is also suitable for various key machine learning optimisation problems including support vector machine (SVM), multiclass SVM, logistic regression, and ridge regression. In particular, the Nestrov's estimate sequence technique to adjust the weight coefficient dynamically and conveniently is adopted. It is applied for training linear SVM from the large training dataset. Experimental results show that the proposed method has a competitive classification performance and faster convergence speed than state-of-the-art algorithms.

Comparing learning to rank techniques in hybrid bug localization

Bug localization is a software development and maintenance activity that aims to find relevant source code entities to be modified so that a specific bug can be fixed on the basis of the given bug report. Information retrieval (IR) techniques have been widely used to locate bugs in recent decades. These techniques mainly use the IR similarity between the bug report and source code entities. In addition to IR similarity, features that are extracted from version history, source code structure, dynamic analysis, and other resources are found to be beneficial for bug localization. The approaches utilizing extra features in IR-based bug localization are called hybrid bug localization. We conduct a short survey of the hybrid bug localization methods that use additional features in addition to IR similarity. We also use Learning to Rank (LtR) techniques to combine the beneficial features to improve bug localization. Learning to Rank is the application of machine learning in the ranking models for information retrieval. We compared eight LtR techniques in bug localization, and the experimental results show that coordinate ascent algorithms without normalization is a suitable LtR technique in bug localization for selected attributes, and it outperforms two state-of-the-art localization approaches for two large projects, Eclipse and SWT.

Optimized Full-Duplex MIMO DF Relaying With Limited Dynamic Range

This paper investigates the transceiver optimization of full-duplex multiple-input multiple-output decode-and-forward (DF) relaying by taking into account the practical factor of limited dynamic range (LDR). Focusing on the maximization of end-to-end achievable rate which involves highly coupled variables, we develop a convergent block coordinate ascent (BCA) algorithm to deal with the complicated optimization problem. In each iteration of BCA, the optimal source beamformer, relay receive beamformer, and destination receive beamformer all admit closed-form solutions. As for the optimization of relay transmit beamformer, we transform the corresponding non-convex maximin problem into a single second-order cone program problem which thus allows an efficient optimal solution. Simulations validate the remarkable performance gain attained by incorporating LDR into the full-duplex DF relaying transceiver optimization.

Binary coordinate ascent: An efficient optimization technique for feature subset selection for machine learning

Feature subset selection (FSS) has been an active area of research in machine learning. A number of techniques have been developed for selecting an optimal or sub-optimal subset of features, because it is a major factor to determine the performance of a machine-learning technique. In this paper, we propose and develop a novel optimization technique, namely, a binary coordinate ascent (BCA) algorithm that is an iterative deterministic local optimization that can be coupled with wrapper or filter FSS. The algorithm searches throughout the space of binary coded input variables by iteratively optimizing the objective function in each dimension at a time. We investigated our BCA approach in wrapper-based FSS under area under the receiver-operating-characteristic (ROC) curve (AUC) criterion for the best subset of features in classification. We evaluated our BCA-based FSS in optimization of features for support vector machine, multilayer perceptron, and Naïve Bayes classifiers with 12 datasets. Our experimental datasets are distinct in terms of the number of attributes (ranging from 18 to 11,340), and the number of classes (binary or multi-class classification). The efficiency in terms of the number of subset evaluations was improved substantially (by factors of 5–37) compared with two popular FSS meta-heuristics, i.e., sequential forward selection (SFS) and sequential floating forward selection (SFFS), while the classification performance for unseen data was maintained.

Multivariate Skew Normal Copula for Non-exchangeable Dependence

The exchangeability assumption on the dependence structure of the multivariate data is restrictive in practical situations where the variables of interest are not likely to be associated to each other in an identical manner. In this paper, we propose a flexible class of multivariate skew normal copulas to model high-dimensional non-exchangeable dependence patterns. The proposed copulas have two sets of parameters capturing non-exchangeable dependence, one for association between the variables and the other for skewness of the variables. In order to efficiently estimate the two sets of parameters, we introduce the block coordinate ascent algorithm. The proposed class of multivariate skew normal copulas is illustrated using a real data set.

Automatic Recognition of Fetal Facial Standard Plane in Ultrasound Image via Fisher Vector.

Acquisition of the standard plane is the prerequisite of biometric measurement and diagnosis during the ultrasound (US) examination. In this paper, a new algorithm is developed for the automatic recognition of the fetal facial standard planes (FFSPs) such as the axial, coronal, and sagittal planes. Specifically, densely sampled root scale invariant feature transform (RootSIFT) features are extracted and then encoded by Fisher vector (FV). The Fisher network with multi-layer design is also developed to extract spatial information to boost the classification performance. Finally, automatic recognition of the FFSPs is implemented by support vector machine (SVM) classifier based on the stochastic dual coordinate ascent (SDCA) algorithm. Experimental results using our dataset demonstrate that the proposed method achieves an accuracy of 93.27% and a mean average precision (mAP) of 99.19% in recognizing different FFSPs. Furthermore, the comparative analyses reveal the superiority of the proposed method based on FV over the traditional methods.

Fast-forming focused spots through a multimode fiber based on an adaptive parallel coordinate algorithm

We propose an adaptive parallel coordinate (APC) algorithm for quickly forming a series of focused spots at a multimode fiber (MMF) output by controlling the MMF input field with a spatial light modulator (SLM). Only passing over the SLM once, we can obtain SLM reflectance to form focused spots on different positions. Compared with the transmission matrix method, our APC does not require iterations and massive calculations. The APC does not require as much access device time as the adaptive sequential coordinate ascent (SCA) algorithm. The experiment results demonstrate that the time taken to form 100 spots with our APC is 1/54th the time with the SCA.

Decomposition Methods for Sparse Matrix Nearness Problems

We discuss three types of sparse matrix nearness problems: given a sparse symmetric matrix, find the matrix with the same sparsity pattern that is closest to it in Frobenius norm and (1) is positive semidefinite, (2) has a positive semidefinite completion, or (3) has a Euclidean distance matrix completion. Several proximal splitting and decomposition algorithms for these problems are presented and their performance is compared on a set of test problems. A key feature of the methods is that they involve a series of projections on small dense positive semidefinite or Euclidean distance matrix cones, corresponding to the cliques in a triangulation of the sparsity graph. The methods discussed include the dual block coordinate ascent algorithm (or Dykstra's method), the dual projected gradient and accelerated projected gradient algorithms, and a primal and a dual application of the Douglas--Rachford splitting algorithm.

Leakage-Aware Energy-Efficient Beamforming for Heterogeneous Multicell Multiuser Systems

Energy-efficient communications has attracted much interest in the research of 5G cellular systems. In this paper, we study energy-efficient coordinated beamforming design for heterogeneous multicell multiuser downlink systems. The considered problem is formulated as maximizing the weighted sum per-cell energy efficiencies (WSPEEMax) subject to predefined per-user target rate demands, maximum leakage interference power constraints, and per-BS transmit power constraints. This formulation is more general than the conventional EE optimization problem and provides a unified way to consider the EE of heterogeneous networks. However, it is hard to tackle due to the weighted sum-of-ratios form of the objective function and the non-convex nature of per-user target rate constraints. To address it, we propose to first transform the original problem into a polynomial form optimization by introducing some auxiliary variables and then further reveal their equivalence in finding the solution. Then, an efficient block coordinate ascent optimization algorithm is developed to solve the equivalent problem by exploiting the concave nature of the considered problem with respect to each optimization variable. To further improve the network EE, we also develop an energy-efficient transmission method for each small-cell network. Finally, extensive numerical results are provided to verify the effectiveness of the proposed schemes and show that both the EE and spectral efficiency (SE) of heterogeneous network can be significantly improved by energy-efficient coordinated multiple-input multiple-output (MIMO) transmission.

Maximum margin multiple-instance feature weighting

Feature weighting is of considerable importance in machine learning due to its effectiveness to highlight relevant components and suppress irrelevant ones. In this paper, we focus on the feature weighting problem in a specific machine learning area: multiple-instance learning, and propose maximum margin multiple-instance feature weighting (M3IFW) to seek large classification margins in the weighted feature space. The designed M3IFW algorithm can be applied to both standard binary-class multiple-instance learning and the corresponding multi-class learning, and we abbreviate them to B-M3IFW (binary-class M3IFW) and M-M3IFW (multi-class M3IFW), respectively. Both B-M3IFW and M-M3IFW contain three kinds of unknown variables, i.e., positive prototypes, classification margins, and weighting coefficients. We utilize the coordinate ascent algorithm to update the three kinds of unknown variables, respectively and iteratively, and then perform classifications in the weighted feature space. Experiments conducted on synthetic and real-world datasets empirically demonstrate the effectiveness of M3IFW in improving classification accuracies.

Patchwork reconstruction with resolution modeling for digital breast tomosynthesis

Digital breast tomosynthesis is a relatively new diagnostic x-ray modality that allows high resolution breast imaging while suppressing interference from overlapping anatomical structures. However, proper visualization of microcalcifications remains a challenge. For the subset of systems considered by the authors, the main cause of deterioration is movement of the x-ray source during exposures. They propose a modified grouped coordinate ascent algorithm that includes a specific acquisition model to compensate for this deterioration. A resolution model based on the movement of the x-ray source during image acquisition is created and combined with a grouped coordinate ascent algorithm. Choosing planes parallel to the detector surface as the groups enables efficient implementation of the position dependent resolution model. In the current implementation, the resolution model is approximated by a Gaussian smoothing kernel. The effect of the resolution model on the iterative reconstruction is evaluated by measuring contrast to noise ratio (CNR) of spherical microcalcifications in a homogeneous background. After this, the new reconstruction method is compared to the optimized filtered backprojection method for the considered system, by performing two observer studies: the first study simulates clusters of spherical microcalcifications in a power law background for a free search task; the second study simulates smooth or irregular microcalcifications in the same type of backgrounds for a classification task. Including the resolution model in the iterative reconstruction methods increases the CNR of microcalcifications. The first observer study shows a significant improvement in detection of microcalcifications (p = 0.029), while the second study shows that performance on a classification task remains the same (p = 0.935) compared to the filtered backprojection method. The new method shows higher CNR and improved visualization of microcalcifications in an observer experiment on synthetic data. Further study of the negative results of the classification task showed performance variations throughout the volume linked to the changing noise structure introduced by the combination of the resolution model and the smoothing prior.