Block Coordinate Descent Scheme Research Articles

Energy-efficient resource allocation in NOMA-integrated V2X networks

In Vehicle-to-Everything (V2X) communication networks, cellular Device-to-Device (D2D) communication can improve the spectrum efficiency, and non-orthogonal multiple access (NOMA) can further enhance the connection density. However, for the cellular D2D communications, the involved execution condition for NOMA may be affected by the additional interference introduced by cellular links in V2X. In this paper, we study the energy-efficient resource allocation problem in cellular D2D-aided V2X networks with NOMA. To efficiently meet the quality of service (QoS) requirements while taking into account the maximum performance from the user’s perspective, we are committed to maximizing the minimum energy efficiency (EE) of each matching link, by jointly optimizing power allocation and spectrum reusing at Vehicle-to-Infrastructure (V2I) links and cellular D2D-based Vehicle-to-Vehicle (V2V) links. Since this problem is mathematically mixed-integer and non-convex, we first develop a two-layer block coordinate descent (BCD) scheme to solve power allocation subproblem. Specifically, the optimal intra-group power allocation strategy is derived in the inner-layer. Then, Dinkelbach and concave–convex procedure (CCCP) are employed to tighten the lower bound of original problem in the outer-layer. Followed by this, we address the remaining spectrum sharing subproblem based on the principle of minimum EE maximization. Simulation results show that the proposed algorithm can achieve excellent performance and outperform the other benchmark schemes significantly.

NOMA-Enabled Computation and Communication Resource Trading for a Multi-User MEC System

In this article, we establish a novel framework for a multi-user mobile edge computing (MEC) network in which a set of users with high downlink rate demands and a set of users with intensive computation tasks can collaborate to achieve a mutually-beneficial scenario such that completion time of the tasks is reduced and the base station (BS) can send more information at a higher rate to the downlink users. Specifically, by leveraging non-orthogonal multiple access (NOMA) for uplink and downlink traffic, the user with the computation task can offload shares of the computation task to the edge cloud and the downlink user. At the same time, this user forwards the information it receives from the BS to the downlink user. In this set up, we jointly optimize the communication resources, computational resources at the edge cloud and user devices, pairings among the two sets of users, the shares of computation tasks, and relay bits to minimize the total task completion time while satisfying downlink users’ incentive requirements. For a network with a single computation demanding user and a single downlink user, the optimal solution to the problem is provided. For a network with multiple users, the problem is non-convex and computationally challenging. Hence, we propose an efficient, low complexity algorithm that utilizes the bottleneck matching algorithm, convex optimization, and the block coordinate descent scheme to obtain a locally-optimal solution. Simulation results demonstrate that, as compared with the state-of-the-art, the total task completion time is greatly reduced (32%–51%), and a large computational energy savings at the edge cloud (38%–55%) is achieved. Simultaneously, the downlink users’ rates improve compared to the orthogonal transmission.

Sparse nonnegative tensor decomposition using proximal algorithm and inexact block coordinate descent scheme

Nonnegative tensor decomposition is a versatile tool for multiway data analysis, by which the extracted components are nonnegative and usually sparse. Nevertheless, the sparsity is only a side effect and cannot be explicitly controlled without additional regularization. In this paper, we investigated the nonnegative CANDECOMP/PARAFAC (NCP) decomposition with the sparse regularization item using l_1-norm (sparse NCP). When high sparsity is imposed, the factor matrices will contain more zero components and will not be of full column rank. Thus, the sparse NCP is prone to rank deficiency, and the algorithms of sparse NCP may not converge. In this paper, we proposed a novel model of sparse NCP with the proximal algorithm. The subproblems in the new model are strongly convex in the block coordinate descent (BCD) framework. Therefore, the new sparse NCP provides a full column rank condition and guarantees to converge to a stationary point. In addition, we proposed an inexact BCD scheme for sparse NCP, where each subproblem is updated multiple times to speed up the computation. In order to prove the effectiveness and efficiency of the sparse NCP with the proximal algorithm, we employed two optimization algorithms to solve the model, including inexact alternating nonnegative quadratic programming and inexact hierarchical alternating least squares. We evaluated the proposed sparse NCP methods by experiments on synthetic, real-world, small-scale, and large-scale tensor data. The experimental results demonstrate that our proposed algorithms can efficiently impose sparsity on factor matrices, extract meaningful sparse components, and outperform state-of-the-art methods.

Federated Block Coordinate Descent Scheme for Learning Global and Personalized Models

In federated learning, models are learned from users’ data that are held private in their edge devices, by aggregating them in the service provider’s “cloud” to obtain a global model. Such global model is of great commercial value in, e.g., improving the customers’ experience. In this paper we focus on two possible areas of improvement of the state of the art. First, we take the difference between user habits into account and propose a quadratic penalty-based formulation, for efficient learning of the global model that allows to personalize local models. Second, we address the latency issue associated with the heterogeneous training time on edge devices, by exploiting a hierarchical structure modeling communication not only between the cloud and edge devices, but also within the cloud. Specifically, we devise a tailored block coordinate descent-based computation scheme, accompanied with communication protocols for both the synchronous and asynchronous cloud settings. We characterize the theoretical convergence rate of the algorithm, and provide a variant that performs empirically better. We also prove that the asynchronous protocol, inspired by multi-agent consensus technique, has the potential for large gains in latency compared to a synchronous setting when the edge-device updates are intermittent. Finally, experimental results are provided that corroborate not only the theory, but also show that the system leads to faster convergence for personalized models on the edge devices, compared to the state of the art.

Hybrid Inexact BCD for Coupled Structured Matrix Factorization in Hyperspectral Super-Resolution

This paper develops a first-order optimization method for coupled structured matrix factorization (CoSMF) problems that arise in the context of hyperspectral super-resolution (HSR) in remote sensing. To best leverage the problem structures for computational efficiency, we introduce a hybrid inexact block coordinate descent (HiBCD) scheme wherein one coordinate is updated via the fast proximal gradient (FPG) method, while another via the Frank-Wolfe (FW) method. The FPG-type methods are known to take less number of iterations to converge, by numerical experience, while the FW-type methods can offer lower per-iteration complexity in certain cases; and we wish to take the best of both. We show that the limit points of this HiBCD scheme are stationary. Our proof treats HiBCD as an optimization framework for a class of multi-block structured optimization problems, and our stationarity claim is applicable not only to CoSMF but also to many other problems. Previous optimization research showed the same stationarity result for inexact block coordinate descent with either FPG or FW updates only. Numerical results indicate that the proposed HiBCD scheme is computationally much more efficient than the state-of-the-art CoSMF schemes in HSR.

Graph Anomaly Detection Using Dictionary Learning

Anomaly detection in networked signals often boils down to identifying an underlying graph structure on which the abnormal occurrence rests on. We investigate the problem of learning graph structure representations using adaptations of dictionary learning aimed at encoding connectivity patterns. In particular, we adapt dictionary learning strategies to the specificity of network topologies and propose new methods that impose Laplacian structure on the dictionaries themselves. In one adaptation we focus on classifying topologies by working directly on the graph Laplacian and cast the learning problem to accommodate its 2D structure. We tackle the same problem by learning dictionaries which consist of vectorized atomic Laplacians, and provide a block coordinate descent scheme to solve the new dictionary learning formulation. Imposing Laplacian structure on the dictionaries is also proposed in an adaptation of the Single Block Orthogonal learning method. Results on synthetic graph datasets comprising different graph topologies confirm the potential of dictionaries to directly represent graph structure information.

Blind Hyperspectral Unmixing Considering the Adjacency Effect

This paper focuses on the blind unmixing technique for analyzing hyperspectral images (HSIs). A joint deconvolution and blind hyperspectral unmixing (DBHU) algorithm is proposed, which is aimed at eliminating the impact of the adjacency effect (AE) on unmixing. In remote sensing imagery, the AE occurs in the presence of atmospheric scattering over a heterogeneous surface. The AE leads to blurring and additional mixing of HSIs and makes it difficult to estimate endmembers and abundances accurately. In this paper, we first model the blurred HSIs by the use of a bilinear mixing model (BMM), where a blurring kernel is used to model the mixing caused by the AE. Based on the BMM, the DBHU problem is formulated as a constrained and biconvex optimization problem. Specifically, the minimum-volume simplex (MVS) is incorporated to deal with the additional mixing caused by the AE, and 3-D total variation (TV) priors are adopted to model the spectral–spatial correlation of the data. In DBHU, the biconvex problem is efficiently solved by a nonstandard application of the alternating direction method of multipliers (ADMM) algorithm, where a block coordinate descent scheme is applied by splitting the original problem into two saddle point subproblems, and then minimizing the subproblems alternately via the ADMM until convergence. The experimental results obtained with both simulated and real data confirm the viability of the proposed algorithm, and DBHU works well, even where both blurring and noise are present in the scene.

Supervised fuzzy partitioning

Centroid-based methods including k-means and fuzzy c-means are known as effective and easy-to-implement approaches to clustering purposes in many applications. However, these algorithms cannot be directly applied to supervised tasks. This paper thus presents a generative model extending the centroid-based clustering approach to be applicable to classification and regression tasks. Given an arbitrary loss function, the proposed approach, termed Supervised Fuzzy Partitioning (SFP), incorporates labels information into its objective function through a surrogate term penalizing the empirical risk. Entropy-based regularization is also employed to fuzzify the partition and to weight features, enabling the method to capture more complex patterns, identify significant features, and yield better performance facing high-dimensional data. An iterative algorithm based on block coordinate descent scheme is formulated to efficiently find a local optimum. Extensive classification experiments on synthetic, real-world, and high-dimensional datasets demonstrate that the predictive performance of SFP is competitive with state-of-the-art algorithms such as SVM and random forest. SFP has a major advantage over such methods, in that it not only leads to a flexible, nonlinear model but also can exploit any convex loss function in the training phase without compromising computational efficiency.

Joint Design of Fronthauling and Hybrid Beamforming for Downlink C-RAN Systems

Hybrid beamforming is known to be a cost-effective and wide-spread solution for a system with large-scale antenna arrays. This paper studies the optimization of the analog and digital components of the hybrid beamforming solution for remote radio heads (RRHs) in a downlink cloud radio access network architecture. Digital processing is carried out at a baseband processing unit (BBU) in the “cloud,” and the precoded baseband signals are quantized prior to transmission to the RRHs via finite-capacity fronthaul links. In this system, we consider two different channel state information (CSI) scenarios: 1) ideal CSI at the BBU and 2) imperfect effective CSI. The optimization of digital beamforming and fronthaul quantization strategies at the BBU as well as analog radio-frequency (RF) beamforming at the RRHs is a coupled problem since the effect of the quantization noise at the receiver depends on the precoding matrices. The resulting joint optimization problem is examined with the goal of maximizing the weighted downlink sum-rate and the network energy efficiency. Fronthaul capacity and per-RRH power constraints are enforced along with constant modulus constraint on the RF beamforming matrices. For the case of perfect CSI, a block coordinate descent scheme is proposed based on the weighted minimum-mean-square-error approach by relaxing the constant modulus constraint of the analog beamformer. Also, we present the impact of imperfect CSI on the weighted sum-rate and network energy efficiency performance, and the algorithm is extended by applying the sample average approximation. The numerical results confirm the effectiveness of the proposed scheme and show that the proposed algorithm is robust to estimation errors.

Robust feature selection via l2,1-norm in finite mixture of regression

Finite mixture of Gaussian regression (FMR) is a widely-used modeling technique in supervised learning problems. In cases where the number of features is large, feature selection is desirable to enhance model interpretability and to avoid overfitting. In this paper, we propose a robust feature selection method via l2,1-norm penalized maximum likelihood estimation (MLE) in FMR, with extension to sparse l2,1 penalty by combining l1-norm with l2,1-norm for increasing flexibility. To solve the non-convex and non-smooth problem of (sparse) penalized MLE in FMR, we develop an new EM-based algorithm for numerical optimization, with combination of block coordinate descent and majorizing-minimization scheme in M-step. We finally apply our method in six simulations and one real dataset to demonstrate its superior performance.

A New First-Order Algorithmic Framework for Optimization Problems with Orthogonality Constraints

In this paper, we consider a class of optimization problems with orthogonality constraints, the feasible region of which is called the Stiefel manifold. Our new framework combines a function value reduction step with a correction step. Different from the existing approaches, the function value reduction step of our algorithmic framework searches along the standard Euclidean descent directions instead of the vectors in the tangent space of the Stiefel manifold, and the correction step further reduces the function value and guarantees a symmetric dual variable at the same time. We construct two types of algorithms based on this new framework. The first type is based on gradient reduction including the gradient reflection (GR) and the gradient projection (GP) algorithms. The other one adopts a columnwise block coordinate descent (CBCD) scheme with a novel idea for solving the corresponding CBCD subproblem inexactly. We prove that both GR/GP with a fixed step size and CBCD belong to our algorithmic framework,...

Hierarchical Sparse Modeling: A Choice of Two Group Lasso Formulations

Demanding sparsity in estimated models has become a routine practice in statistics. In many situations, we wish to require that the sparsity patterns attained honor certain problem-specific constraints. Hierarchical sparse modeling (HSM) refers to situations in which these constraints specify that one set of parameters be set to zero whenever another is set to zero. In recent years, numerous papers have developed convex regularizers for this form of sparsity structure, which arises in many areas of statistics including interaction modeling, time series analysis, and covariance estimation. In this paper, we observe that these methods fall into two frameworks, the group lasso (GL) and latent overlapping group lasso (LOG), which have not been systematically compared in the context of HSM. The purpose of this paper is to provide a side-by-side comparison of these two frameworks for HSM in terms of their statistical properties and computational efficiency. We call special attention to GL's more aggressive shrinkage of parameters deep in the hierarchy, a property not shared by LOG. In terms of computation, we introduce a finite-step algorithm that exactly solves the proximal operator of LOG for a certain simple HSM structure; we later exploit this to develop a novel path-based block coordinate descent scheme for general HSM structures. Both algorithms greatly improve the computational performance of LOG. Finally, we compare the two methods in the context of covariance estimation, where we introduce a new sparsely-banded estimator using LOG, which we show achieves the statistical advantages of an existing GL-based method but is simpler to express and more efficient to compute.

The Iterative Reweighted Mixed-Norm Estimate for Spatio-Temporal MEG/EEG Source Reconstruction.

Source imaging based on magnetoencephalography (MEG) and electroencephalography (EEG) allows for the non-invasive analysis of brain activity with high temporal and good spatial resolution. As the bioelectromagnetic inverse problem is ill-posed, constraints are required. For the analysis of evoked brain activity, spatial sparsity of the neuronal activation is a common assumption. It is often taken into account using convex constraints based on the l1-norm. The resulting source estimates are however biased in amplitude and often suboptimal in terms of source selection due to high correlations in the forward model. In this work, we demonstrate that an inverse solver based on a block-separable penalty with a Frobenius norm per block and a l0.5-quasinorm over blocks addresses both of these issues. For solving the resulting non-convex optimization problem, we propose the iterative reweighted Mixed Norm Estimate (irMxNE), an optimization scheme based on iterative reweighted convex surrogate optimization problems, which are solved efficiently using a block coordinate descent scheme and an active set strategy. We compare the proposed sparse imaging method to the dSPM and the RAP-MUSIC approach based on two MEG data sets. We provide empirical evidence based on simulations and analysis of MEG data that the proposed method improves on the standard Mixed Norm Estimate (MxNE) in terms of amplitude bias, support recovery, and stability.

A Symmetric Sparse Representation Based Band Selection Method for Hyperspectral Imagery Classification

A novel Symmetric Sparse Representation (SSR) method has been presented to solve the band selection problem in hyperspectral imagery (HSI) classification. The method assumes that the selected bands and the original HSI bands are sparsely represented by each other, i.e., symmetrically represented. The method formulates band selection into a famous problem of archetypal analysis and selects the representative bands by finding the archetypes in the minimal convex hull containing the HSI band points (i.e., one band corresponds to a band point in the high-dimensional feature space). Without any other parameter tuning work except the size of band subset, the SSR optimizes the band selection program using the block-coordinate descent scheme. Four state-of-the-art methods are utilized to make comparisons with the SSR on the Indian Pines and PaviaU HSI datasets. Experimental results illustrate that SSR outperforms all four methods in classification accuracies (i.e., Average Classification Accuracy (ACA) and Overall Classification Accuracy (OCA)) and three quantitative evaluation results (i.e., Average Information Entropy (AIE), Average Correlation Coefficient (ACC) and Average Relative Entropy (ARE)), whereas it takes the second shortest computational time. Therefore, the proposed SSR is a good alternative method for band selection of HSI classification in realistic applications.

Regularized Discrete Optimal Transport

This article introduces a generalization of the discrete optimal transport, with applications to color image manipulations. This new formulation includes a relaxation of the mass conservation constraint and a regularization term. These two features are crucial for image processing tasks where one must take into account families of multimodal histograms with large mass variation across modes. The corresponding relaxed and regularized transportation problem is the solution of a convex optimization problem. Depending on the regularization used, this minimization can be solved using standard linear programming methods or first order proximal splitting schemes. The resulting transportation plan can be used as a color transfer map, which is robust to mass variation across image color palettes. Furthermore, the regularization of the transport plan helps remove colorization artifacts due to noise amplification. We also extend this framework to compute the barycenter of distributions. The barycenter is the solution of an optimization problem, which is separately convex with respect to the barycenter and the transportation plans, but not jointly convex. A block coordinate descent scheme converges to a stationary point of the energy. We show that the resulting algorithm can be used for color normalization across several images. The relaxed and regularized barycenter defines a common color palette for those images. Applying color transfer toward this average palette performs a color normalization of the input images.

A continuous characterization of the maximum-edge biclique problem

The problem of finding large complete subgraphs in bipartite graphs (that is, bicliques) is a well-known combinatorial optimization problem referred to as the maximum-edge biclique problem (MBP), and has many applications, e.g., in web community discovery, biological data analysis and text mining. In this paper, we present a new continuous characterization for MBP. Given a bipartite graph $$G$$ , we are able to formulate a continuous optimization problem (namely, an approximate rank-one matrix factorization problem with nonnegativity constraints, R1N for short), and show that there is a one-to-one correspondence between (1) the maximum (i.e., the largest) bicliques of $$G$$ and the global minima of R1N, and (2) the maximal bicliques of $$G$$ (i.e., bicliques not contained in any larger biclique) and the local minima of R1N. We also show that any stationary points of R1N must be close to a biclique of $$G$$ . This allows us to design a new type of biclique finding algorithm based on the application of a block-coordinate descent scheme to R1N. We show that this algorithm, whose algorithmic complexity per iteration is proportional to the number of edges in the graph, is guaranteed to converge to a biclique and that it performs competitively with existing methods on random graphs and text mining datasets. Finally, we show how R1N is closely related to the Motzkin---Strauss formalism for cliques.