Rank-constrained Optimization Research Articles

Exact penalization for cardinality and rank-constrained optimization problems via partial regularization

In this paper, we consider a class of constrained optimization problems whose constraints involve a cardinality or rank constraint. The penalty formulation based on a partial regularization has recently been promoted in the literature to approximate these problems, which usually outperforms the penalty formulation based on a full regularization in terms of solution quality. Nevertheless, the relation between the penalty formulation with a partial regularizer and the original problem was not much studied yet. Under some suitable assumptions, we show that the penalty formulation based on a partial regularization is an exact reformulation of the original problem in the sense that they both share the same global minimizers. We also show that a local minimizer of the original problem is that of the penalty reformulation. These results provide some theoretical justification for the often-observed superior performance of the penalty model based on a partial regularizer over a corresponding full regularizer.

A global exact penalty for rank-constrained optimization problem and applications

A global exact penalty for rank-constrained optimization problem and applications

Quantum-Inspired Hierarchy for Rank-Constrained Optimization

Many problems in information theory can be reduced to optimizations over matrices, where the rank of the matrices is constrained. We establish a link between rank-constrained optimization and the theory of quantum entanglement. More precisely, we prove that a large class of rank-constrained semidefinite programs can be written as a convex optimization over separable quantum states and, consequently, we construct a complete hierarchy of semidefinite programs for solving the original problem. This hierarchy not only provides a sequence of certified bounds for the rank-constrained optimization problem, but also gives pretty good and often exact values in practice when the lowest level of the hierarchy is considered. We demonstrate that our approach can be used for relevant problems in quantum information processing, such as the optimization over pure states, the characterization of mixed unitary channels and faithful entanglement, and quantum contextuality, as well as in classical information theory including the maximum cut problem, pseudo-Boolean optimization, and the orthonormal representation of graphs. Finally, we show that our ideas can be extended to rank-constrained quadratic and higher-order programming.

Stabilization of two-dimensional mixed continuous-discrete-time systems via dynamic output feedback with application to iterative learning control design

This paper investigates the asymptotic stabilization of the 2-D continuous-discrete-time systems via dynamic output feedback. The problem is formulated in a general framework, where the system and controller are both described by the Roesser model. By rigorous mathematical works, a linear matrix inequality (LMI) condition is established to ensure the asymptotic stability of the closed-loop system. Using this LMI condition, the problem of determining the stabilizing controller parameters is converted to a rank minimization problem (RMP) with an LMI constraint, or equivalently to a rank-constrained optimization problem (RCOP). When this RMP (or equivalently this RCOP) has a solution, then the controller parameters are obtained uniquely by an explicit formula in terms of this solution. Moreover, it is shown that one can transform the problem of one-dimensional continuous-time iterative learning control (ILC) design to a 2-D stabilization problem, and consequently solve it using the presented two-dimensional scheme. Some numerical examples are given to illustrate the proposed method.

Efficient low-order system identification from low-quality step response data with rank-constrained optimization

In the presence of low-quality industrial process data, generic step response identification methods typically show unsatisfactory performance and heavily rely on manual intervention of technical personnel. This erects obvious obstacles for the advancement of intelligent manufacturing in process industries. To address these challenges, we propose a novel rank-constrained optimization approach to low-order system identification from step response data, which yields much more accurate and robust estimates than existing modeling methods. By exploiting the inherent low-rank structure of the Hankel matrix of ideal step response, parameters of a low-order process can be accurately recovered by solving a rank-constrained program, which effectively bypasses the two-step procedure in some state-of-the-art algorithms involving significant error accumulation. The alternating direction method of multipliers is adopted to effectively solve the nonconvex error minimization problem and circumvent poor local optima. Case studies on both numerical examples and industrial datasets demonstrate that, the proposed method not only gives much better modeling accuracy, but also secures reliable and robust estimates even for raw low-quality industrial data. This is particularly helpful for automated execution of the identification routine without human intervention, with success percentage over 99% that is remarkably higher than the state-of-the-art.

Frequency Response Function identification for multivariable motion control: Optimal experiment design with element-wise constraints

Frequency Response Functions (FRFs) are essential in mechatronic systems and its application ranges from system design and validation to controller design and diagnostics. The aim of this paper is to optimally design experiments for FRF identification of multivariable motion systems subject to element-wise power constraints. A multivariable excitation design framework is established that explicitly addresses the frequency-wise directionality of the system to be identified. The design problem involves solving a rank-constrained optimization problem, which is non-convex and NP-hard in most cases. Two algorithms to solving this problem approximately are presented that rely on a convex (semi-definite) relaxation of the original problem. Additionally, exact solutions for several special cases are presented. The two algorithms are shown to overcome the limitations of traditional excitation design. This is confirmed by experimental results from a 7 × 8 wafer stage setup, which show a significant improvement of the FRF quality using the proposed techniques over traditional design approaches.

Ranking from Crowdsourced Pairwise Comparisons via Smoothed Riemannian Optimization

Social Internet of Things has recently become a promising paradigm for augmenting the capability of humans and devices connected in the networks to provide services. In social Internet of Things network, crowdsourcing that collects the intelligence of the human crowd has served as a powerful tool for data acquisition and distributed computing. To support critical applications (e.g., a recommendation system and assessing the inequality of urban perception), in this article, we shall focus on the collaborative ranking problems for user preference prediction from crowdsourced pairwise comparisons. Based on the Bradley--Terry--Luce (BTL) model, a maximum likelihood estimation (MLE) is proposed via low-rank approach in order to estimate the underlying weight/score matrix, thereby predicting the ranking list for each user. A novel regularized formulation with the smoothed surrogate of elementwise infinity norm is proposed in order to address the unique challenge of the coupled the non-smooth elementwise infinity norm constraint and non-convex low-rank constraint in the MLE problem. We solve the resulting smoothed rank-constrained optimization problem via developing the Riemannian trust-region algorithm on quotient manifolds of fixed-rank matrices, which enjoys the superlinear convergence rate. The admirable performance and algorithmic advantages of the proposed method over the state-of-the-art algorithms are demonstrated via numerical results. Moreover, the proposed method outperforms state-of-the-art algorithms on large collaborative filtering datasets in both success rate of inferring preference and normalized discounted cumulative gain.

Joint Optimization of Dimension Assignment and Compression in Distributed Estimation Fusion

This paper studies linear distributed estimation of an unknown random parameter vector in a bandwidth-constrained multisensor network. To meet the bandwidth limitations, each sensor converts its observation into a low-dimensional datum via a suitable linear transformation. Then, the fusion center estimates the parameter vector by linearly combining all the received low-dimensional data, aiming at minimizing the estimation mean square error. The main purpose of this paper is to jointly determine the compression dimension of each sensor (referred to as dimension assignment) and design the corresponding compression matrix when the total compression dimensions is limited. Such a joint design problem can be formulated as a rank-constrained optimization problem and it is shown to be NP-hard for the first time. In addition, successive quadratic upper-bound minimization (SQUM), SQUM-block coordinate descent (SQUM-BCD) and nuclear norm regularization (NNR) methods are developed to solve it approximately. Furthermore, we show that any accumulation point of the sequence generated by the SQUM method satisfies the Karush-Kuhn-Tucker conditions of the rank-constrained optimization problem, and the Phase II algorithm of the SQUM-BCD and NNR methods (both are two-phase algorithms and have the same Phase II algorithm) guarantees convergence at least to a stationary point. Numerical experiments illustrate the advantages of the proposed methods compared with the existing method.

Identification of the dynamics of time-varying phase aberrations from time histories of the point-spread function.

To optimally compensate for time-varying phase aberrations with adaptive optics, a model of the dynamics of the aberrations is required to predict the phase aberration at the next time step. We model the time-varying behavior of a phase aberration, expressed in Zernike modes, by assuming that the temporal dynamics of the Zernike coefficients can be described by a vector-valued autoregressive (VAR) model. We propose an iterative method based on a convex heuristic for a rank-constrained optimization problem, to jointly estimate the parameters of the VAR model and the Zernike coefficients from a time series of measurements of the point-spread function (PSF) of the optical system. By assuming the phase aberration is small, the relation between aberration and PSF measurements can be approximated by a quadratic function. As such, our method is a blind identification method for linear dynamics in a stochastic Wiener system with a quadratic nonlinearity at the output and a phase retrieval method that uses a time-evolution-model constraint and a single image at every time step.

Joint Beamforming and Power Allocation in Downlink NOMA Multiuser MIMO Networks

In this paper, a novel joint design of beamforming and power allocation is proposed for a multi-cell multiuser multiple-input multiple-output (MIMO) non-orthogonal multiple access (NOMA) network. In this network, base stations (BSs) adopt coordinated multipoint (CoMP) for downlink transmission. We study a new scenario where the users are divided into two groups according to their quality-of-service (QoS) requirements, rather than their channel qualities as investigated in the literature. Our proposed joint design aims to maximize the sum-rate of the users in one group with the best-effort while guaranteeing the minimum required target rates of the users in the other group. The joint design is formulated as a non-convex NP-hard problem. To make the problem tractable, a series of transformations are adopted to simplify the design problem. Then, an iterative suboptimal resource allocation algorithm based on successive convex approximation is proposed. In each iteration, a rank-constrained optimization problem is solved optimally via semidefinite program relaxation. Numerical results reveal that the proposed scheme offers significant sum-rate gains compared to the existing schemes and converges fast to a suboptimal solution.

Flexible Low-Rank Statistical Modeling with Missing Data and Side Information

We explore a general statistical framework for low-rank modeling of matrix-valued data, based on convex optimization with a generalized nuclear norm penalty. We study several related problems: the usual low-rank matrix completion problem with flexible loss functions arising from generalized linear models; reduced-rank regression and multi-task learning; and generalizations of both problems where side information about rows and columns is available, in the form of features or smoothing kernels. We show that our approach encompasses maximum a posteriori estimation arising from Bayesian hierarchical modeling with latent factors, and discuss ramifications of the missing-data mechanism in the context of matrix completion. While the above problems can be naturally posed as rank-constrained optimization problems, which are nonconvex and computationally difficult, we show how to relax them via generalized nuclear norm regularization to obtain convex optimization problems. We discuss algorithms drawing inspiration from modern convex optimization methods to address these large scale convex optimization computational tasks. Finally, we illustrate our flexible approach in problems arising in functional data reconstruction and ecological species distribution modeling.

Identification of low-order models using Laguerre basis function expansions

In this paper, a method for identification of low-order output-error models expressed in terms of Laguerre basis functions is presented. The identification problem is formulated as a rank-constrained optimization problem in the Laguerre domain, which can be solved efficiently using nuclear norm regularization. Since the identified models are of low order, a minimal state-space realization of the system can be obtained without the use of additional approximative model-order reduction steps.

Sparse Memoryless LQR Design for Uncertain Linear Time-Delay Systems

The sparse memoryless LQR design problem is formulated for uncertain linear time-delay systems. In such a problem, the goal is to minimize a quadratic cost supplemented by sparsity-promoting term (weighted-ℓ1 in our case) subject to stability of closed-loop system under norm-bounded uncertainty. It is shown that such an optimization problem can be reformulated as a rank-constrained optimization problem which consists of convex constraints except one rank constraint. Utilizing the bi-linear rank penalty technique, the sparse memoryless LQR is designed. Numerous numerical results depict that there exists a trade-off between time-delay and sparsification quality. In addition, the larger time-delay, the poorer performance-sparsity trade-off is observed.

Rank-constrained optimization and its applications

This paper investigates an iterative approach to solve the general rank-constrained optimization problems (RCOPs) defined to optimize a convex objective function subject to a set of convex constraints and rank constraints on unknown rectangular matrices. In addition, rank minimization problems (RMPs) are introduced and equivalently transformed into RCOPs by introducing a quadratic matrix equality constraint. The rank function is discontinuous and nonconvex, thus the general RCOPs are classified as NP-hard in most of the cases. An iterative rank minimization (IRM) method, with convex formulation at each iteration, is proposed to gradually approach the constrained rank. The proposed IRM method aims at solving RCOPs with rank inequalities constrained by upper or lower bounds, as well as rank equality constraints. Proof of the convergence to a local minimizer with at least a sublinear convergence rate is provided. Four representative applications of RCOPs and RMPs, including system identification, output feedback stabilization, and structured H2 controller design problems, are presented with comparative simulation results to verify the feasibility and improved performance of the proposed IRM method.

A novel representation of rank constraints for real matrices

We present a novel representation of rank constraints for non-square real matrices. We establish relationships with existing results and show that these are particular cases of our representation. One of these cases is a representation of the ℓ0 pseudo-norm, which is used in sparse representation problems. Finally, we describe how our representation can be included in rank-constrained optimization and in rank-minimization problems.

Application of Rank-Constrained Optimisation to Nonlinear System Identification

Nonlinear System Identification has a rich history spanning at least 5 decades. A very flexible approach to this problem depends upon the use of Volterra series expansions. Related work includes Hammerstein models, where a static nonlinearity is followed by a linear dynamical system, and Wiener models, where a static nonlinearity is inserted after a linear dynamical model. A problem with these methods is that they inherently depend upon series type expansions and hence it is dificult to know which terms should be included. In this paper we present a possible solution to this problem using recent results on rank-constrained optimization. Simulation results are included to illustrate the eficacy of the proposed strategy.

On the spectral efficiency of full-duplex small cell wireless systems

We investigate the spectral efficiency of full-duplex small cell wireless systems, in which a full-duplex capable base station (BS) is designed to send/receive data to/from multiple halfduplex users on the same system resources. The major hurdle for designing such systems is due to the self-interference at the BS and co-channel interference among users. Hence, we consider a joint beamformer design to maximize the spectral efficiency subject to certain power constraints. The design problem is first formulated as a rank-constrained optimization one, and the rank relaxation method is then applied. However the relaxed problem is still nonconvex, and thus optimal solutions are hard to find. Herein, we propose two provably convergent algorithms to obtain suboptimal solutions. Based on the concept of the difference of convex functions programming, we approximate the design problem by a determinant maximization program in each iteration of the first algorithm. The second method is built upon the sequential parametric convex approximation method, which allows us to transform the relaxed problem into a semidefinite program in each iteration. Extensive numerical experiments under small cell setups illustrate that the full-duplex system with the proposed algorithms can achieve a large gain over the half-duplex one.

A Rank-Constrained Optimization approach: Application to Factor Analysis

In this paper, we present a general method for rank-constrained optimization. We use an iterative convex optimization procedure where it is possible to include any extra convex constraints. The proposed approach has potential application in several areas. We focus on the problem of Factor Analysis. In this case, our approach provides sufficient flexibility to handle correlated errors. The benefits of the method is demonstrated via a simulation study.

Robust Adaptive Array Beamforming With Subspace Steering Vector Uncertainties

In this letter, the steering vector of the signal-of-interest (SOI) is assumed to belong to a given linear subspace but the associated coordinates are otherwise unknown. In the case where the signal subspace is poorly estimated, a rank-constrained optimization problem is formulated in which the signal subspace is forced to intersect with the known linear subspace (where the SOI steering vector is located). Such problem can be reformulated as two subproblems via the variable alternation method. Also, the analytical solutions of these two subproblems can be found. Numerical results demonstrate that the proposed method can be regarded as an improved subspace-based method, which lowers the requirements in terms of snapshot number or signal-to-noise ratio (SNR) to outperform the diagonal-loading-type methods as compared with the traditional signal-subspace projection method.

Rank-Constrained Schur-Convex Optimization With Multiple Trace/Log-Det Constraints

Rank-constrained optimization problems have received an increasing intensity of interest recently, because many optimization problems in communications and signal processing applications can be cast into a rank-constrained optimization problem. However, due to the nonconvex nature of rank constraints, a systematic solution to general rank-constrained problems has remained open for a long time. In this paper, we focus on a rank-constrained optimization problem with a Schur-convex/concave objective function and multiple trace/log-determinant constraints. We first derive a structural result on the optimal solution of the rank-constrained problem using majorization theory. Based on the solution structure, we transform the rank-constrained problem into an equivalent problem with a unitary constraint. After that, we derive an iterative projected steepest descent algorithm which converges to a local optimal solution. Furthermore, we shall show that under some special cases, we can derive a closed-form global optimal solution. The numerical results show the superior performance of our proposed technique over the baseline schemes.