Non-negative Least-squares Problem Research Articles

Systems of Linear Equations with Non-Negativity Constraints: Hyper-Rectangle Cover Theory and Its Applications

In this paper, a novel hyper-rectangle cover theory is developed. Two important concepts, the cover order and the cover length, are introduced. We construct a specific échelon form of the matrix in the same manner as that employed to determine the rank of the matrix to obtain the cover order of any given matrix. Using the properties of the cover order, we obtain the necessary and sufficient conditions for the existence and uniqueness of the solutions for linear equations system with non-negativity constraints on variables for both homogeneous and nonhomogeneous cases. In addition, we apply the cover theory to analyze some typical problems in linear algebra and optimization with non-negativity constraints on variables, including linear programming (LP) problems and non-negative least squares (NNLS) problems. For LP problems, the three possible behaviours of the solutions are studied through cover theory. On the other hand, we develop a method to obtain the cover length of the covered variable. In this process, we discover the relationship between the cover length determination problem and the NNLS problem. This enables us to obtain an analytical optimal value for the NNLS problem.

Structured channel covariance estimation from limited samples for large antenna arrays

In massive multiuser multiple antenna systems, the knowledge of the users’ channel covariance matrix is crucial for minimum mean square error channel estimation in the uplink and it plays an important role in several multiuser beamforming schemes in the downlink. Due to the large number of base station antennas, accurate covariance estimation is challenging especially in the case where the number of samples is limited and thus comparable to the channel vector dimension. As a result, the standard sample covariance estimator may yield a too large estimation error which in turn may yield significant system performance degradation with respect to the ideal channel covariance knowledge case. To address such problem, we propose a method based on a parametric representation of the channel angular scattering function. The proposed parametric representation includes a discrete specular component which is addressed using the well-known MUltiple SIgnal Classification (MUSIC) method, and a diffuse scattering component, which is modeled as the superposition of suitable dictionary functions. To obtain the representation parameters, we propose two methods, where the first solves a nonnegative least-squares problem and the second maximizes the likelihood function using expectation–maximization. Our simulation results show that the proposed methods outperform the state of the art with respect to various estimation quality metrics and different sample sizes.

Fast Sparse Non-Negative Least Squares via ADMM for High Resolution DOA Estimation

High-resolution and low-complexity direction of arrival (DOA) estimation deserves a warm welcome in a real radar sensor. However, achieving this goal is nontrivial. For the DOA estimation of the uniform linear array (ULA), we offer an efficient approach to reach a good compromise between complexity, resolution and accuracy. Under the covariance fitting framework, we construct a sparse non-negative least squares (NNLS) that is solved by the alternating direction method of multipliers (ADMM) at a low-computational complexity. The regularization item involved in the ADMM procedure yields an inherent improvement on robustness. Moreover, a large-scale matrix inversion that is a computationally expensive operation in the ADMM procedure is recursively computed by exploiting the diagonal matrix property of the regularization item, resulting in a reduction on complexity. In addition, the pre-estimation of the noise variance further reduces the scale of the sparse NNLS problem and alleviates the ill-conditioning effect to some extent. Compared to several sparse covariance fitting methods, both the simulation data and the 77 GHz radar-measured data demonstrate that the proposed algorithm has the good resolution, robustness, and accuracy performance with lower computationally cost.

An index search method based inner-outer iterative algorithm for solving nonnegative least squares problems

We propose an index search method (ISM) for solving nonnegative least squares problems (NNLS). This method uses inner and outer schemes to find the index set corresponding to the nonzero component of the optimal solution. The outer iteration updates the approximate index set such that the objective value of the sequence generated by ISM is monotonically decreasing. Hence, the index set generated by ISM does not repeat, and the optimal solution can be achieved with finite iteration steps. Some normal equations need to be solved in the inner iteration, which is the dominant computational complexity in ISM. Numerical experiments are provided to support the theoretical results.

Massive Machine-Type Communication Pilot-Hopping Sequence Detection Architectures Based on Non-Negative Least Squares for Grant-Free Random Access

User activity detection in grant-free random access massive machine type communication (mMTC) using pilot-hopping sequences can be formulated as solving a non-negative least squares (NNLS) problem. In this work, two architectures using different algorithms to solve the NNLS problem is proposed. The algorithms are implemented using a fully parallel approach and fixed-point arithmetic, leading to high detection rates and low power consumption. The first algorithm, fast projected gradients, converges faster to the optimal value. The second algorithm, multiplicative updates, is partially implemented in the logarithmic domain, and provides a smaller chip area and lower power consumption. For a detection rate of about one million detections per second, the chip area for the fast algorithm is about 0.7 mm <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> compared to about 0.5 mm <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> for the multiplicative algorithm when implemented in a 28 nm FD-SOI standard cell process at 1 V power supply voltage. The energy consumption is about 300 nJ/detection for the fast projected gradient algorithm using 256 iterations, leading to a convergence close to the theoretical. With 128 iterations, about 250 nJ/detection is required, with a detection performance on par with 192 iterations of the multiplicative algorithm for which about 100 nJ/detection is required.

Non-negative Least Squares Approach to Quantification of 1H Nuclear Magnetic Resonance Spectra of Human Urine.

Because of its quantitative character and capability for high-throughput screening, 1H nuclear magnetic resonance (NMR) spectroscopy is used extensively in the profiling of biofluids such as urine and blood plasma. However, the narrow frequency bandwidth of 1H NMR spectroscopy leads to a severe overlap of the spectra of components present in the complex mixtures such as biofluids. Therefore, 1H NMR-based metabolomics analysis is focused on targeted studies related to concentrations of the small number of metabolites. Here, we propose a library-based approach to quantify proportions of overlapping metabolites from 1H NMR mixture spectra. The method boils down to the linear non-negative least squares (NNLS) problem, whereas proportions of the pure components contained in the library stand for the unknowns. The method is validated on an estimation of the proportions of (i) the 78 pure spectra, presumably related to type 2 diabetes mellitus (T2DM), from their synthetic linear mixture; (ii) metabolites present in 62 1H NMR spectra of urine of subjects with T2DM and 62 1H NMR spectra of urine of control subjects. In both cases, the in-house library of 210 pure component 1H NMR spectra represented the design matrix in the related NNLS problem. The proposed method pinpoints 63 metabolites that in a statistically significant way discriminate the T2DM group from the control group and 46 metabolites discriminating control from the T2DM group. For several T2DM-discriminative metabolites, we prove their presence by independent analytical determination or by pointing out the corresponding findings in the published literature.

Exact Complexity Certification of a Nonnegative Least-Squares Method for Quadratic Programming

In this letter we propose a method to exactly certify the complexity of an active-set method which is based on reformulating strictly convex quadratic programs to nonnegative least-squares problems. The exact complexity of the method is determined by proving the correspondence between the method and a standard primal active-set method for quadratic programming applied to the dual of the quadratic program to be solved. Once this correspondence has been established, a complexity certification method which has already been established for the primal active-set method is used to also certify the complexity of the nonnegative least-squares method. The usefulness of the proposed method is illustrated on a multi-parametric quadratic program originating from model predictive control of an inverted pendulum.

Safe Feature Elimination for Non-negativity Constrained Convex Optimization

Inspired by recent work on safe feature elimination for $1$-norm regularized least-squares, we develop strategies to eliminate features from convex optimization problems with non-negativity constraints. Our strategy is safe in the sense that it will only remove features/coordinates from the problem when they are guaranteed to be zero at a solution. To perform feature elimination we use an accurate, but not optimal, primal-dual feasible pair, making our methods robust and able to be used on ill-conditioned problems. We supplement our feature elimination problem with a method to construct an accurate dual feasible point from an accurate primal feasible point; this allows us to use a first-order method to find an accurate primal feasible point, then use that point to construct an accurate dual feasible point and perform feature elimination. Under reasonable conditions, our feature elimination strategy will eventually eliminate all zero features from the problem. As an application of our methods we show how safe feature elimination can be used to robustly certify the uniqueness of non-negative least-squares (NNLS) problems. We give numerical examples on a well-conditioned synthetic NNLS problem and a on set of 40000 extremely ill-conditioned NNLS problems arising in a microscopy application.

The Weighted Nonnegative Least-Squares Problem with Implicitly Characterized Points

In “The Weighted Nonnegative Least-Squares Problem with Implicitly Characterized Points,” A. Fattahi, S. Dasu, and R. Ahmadi study a new variation of nonnegative least-squares problem when the disc...

Embedded Model Predictive Control for Enhancing Tracking Performance of a Ball-and-Plate System

Implementing model predictive control (MPC) on a limited-performance microcontroller has always been a great challenge. In this paper, we propose an MPC implementation method that can run on a microcontroller by improving the calculation efficiency and address applied to the ball-and-plate system to enhance tracking performance. First, we convert the MPC quadratic programming problem, which causes difficulties in implementing MPC, into an equivalent nonnegative least-squares problem so that we can build the solver in a C program easily. Subsequently, we separate the offline calculation from the online process, which drastically reduces the calculation time. Finally, we implement the proposed MPC in a C program to run it on a Nucleo-32 microcontroller, apply it to a tracking control problem using a laboratory-built ball-and-plate system, and then explain the improved tracking performance compared with the conventional control methods.

Nesterov-Based Alternating Optimization for Nonnegative Tensor Factorization: Algorithm and Parallel Implementation

We consider the problem of nonnegative tensor factorization. Our aim is to derive an efficient algorithm that is also suitable for parallel implementation. We adopt the alternating optimization framework and solve each matrix nonnegative least-squares problem via a Nesterov-type algorithm for strongly convex problems. We describe a parallel implementation of the algorithm and measure the attained speedup in a multicore computing environment. It turns out that the derived algorithm is a competitive candidate for the solution of very large-scale dense nonnegative tensor factorization problems.

Dimensional hyper-reduction of nonlinear finite element models via empirical cubature

We present a general framework for the dimensional reduction, in terms of number of degrees of freedom as well as number of integration points (“hyper-reduction”), of nonlinear parameterized finite element (FE) models. The reduction process is divided into two sequential stages. The first stage consists in a common Galerkin projection onto a reduced-order space, as well as in the condensation of boundary conditions and external forces. For the second stage (reduction in number of integration points), we present a novel cubature scheme that efficiently determines optimal points and associated positive weights so that the error in integrating reduced internal forces is minimized. The distinguishing features of the proposed method are: (1) The minimization problem is posed in terms of orthogonal basis vector (obtained via a partitioned Singular Value Decomposition) rather that in terms of snapshots of the integrand. (2) The volume of the domain is exactly integrated. (3) The selection algorithm need not solve in all iterations a nonnegative least-squares problem to force the positiveness of the weights. Furthermore, we show that the proposed method converges to the absolute minimum (zero integration error) when the number of selected points is equal to the number of internal force modes included in the objective function. We illustrate this model reduction methodology by two nonlinear, structural examples (quasi-static bending and resonant vibration of elastoplastic composite plates). In both examples, the number of integration points is reduced three order of magnitudes (with respect to FE analyses) without significantly sacrificing accuracy.

FAST NON-NEGATIVE LEAST-SQUARES LEARNING IN THE RANDOM NEURAL NETWORK

The random neural network is a biologically inspired neural model where neurons interact by probabilistically exchanging positive and negative unit-amplitude signals that has superior learning capabilities compared to other artificial neural networks. This paper considers non-negative least squares supervised learning in this context, and develops an approach that achieves fast execution and excellent learning capacity. This speedup is a result of significant enhancements in the solution of the non-negative least-squares problem which regard (a) the development of analytical expressions for the evaluation of the gradient and objective functions and (b) a novel limited-memory quasi-Newton solution algorithm. Simulation results in the context of optimizing the performance of a disaster management problem using supervised learning verify the efficiency of the approach, achieving two orders of magnitude execution speedup and improved solution quality compared to state-of-the-art algorithms.

Interior point gradient algorithm for totally nonnegative least-squares problems in inequality sense

In this paper we study a scaled interior-point gradient algorithm for totally nonnegative least-squares problems in the inequality sense. The global convergence of the algorithm is established. Finally, extensive computational comparison with the so-called generalized Newton algorithm is given showing the superior performance of the gradient-based algorithm.

A non-monotonic method for large-scale non-negative least squares

We present a new algorithm for solving the non-negative least-squares (NNLS) problem. Our algorithm extends the unconstrained quadratic optimization algorithm of Barzilai and Borwein (BB) [J. Barzilai and J. M. Borwein; Two-Point Step Size Gradient Methods. IMA J. Numer. Anal. 1988.] to handle nonnegativity constraints. Our extension differs from other constrained BB variants in simple but crucial aspects, the most notable being our modification to the BB stepsize itself. Our stepsize computation takes into account the nonnegativity constraints, and is further refined by a stepsize scaling strategy. These changes, in combination with orthogonal projections onto the nonnegative orthant, yield an effective NNLS algorithm. We compare our algorithm with several competing approaches, including established bound-constrained solvers, popular BB-based methods, and also a specialised NNLS algorithm. On several synthetic and real-world datasets our method displays highly competitive empirical performance.

Linear inequalities via least squares

The Gaussian elimination method is usually used in solving problems related to systems of linear inequalities. The present review paper describes the application of the least-squares method to studying problems connected with linear inequalities (like redundant inequalities, theorems of alternative, mathematical programming). The minimum norm solution to the system of linear inequalities is found by solving a non-negative least-squares (NNLS) problem. A linear programming (LP) problem is transformed to the system of inequalities in several ways. By solving the corresponding NNLS problem an initial solution to the LP problem is found. The main ideas are explained by simple examples.

A Matrix-Free Preconditioner for Sparse Symmetric Positive Definite Systems and Least-Squares Problems

We analyze and discuss matrix-free and limited memory preconditioners for sparse symmetric positive definite systems and normal equations of large and sparse least-squares problems. The preconditioners are based on a partial Cholesky factorization and can be coupled with a deflation strategy. The construction of the preconditioners requires only matrix-vector products, is breakdown-free, and does not need to form the matrix. The memory requirements of the preconditioners are defined in advance, and they do not depend on the number of nonzero entries in the matrix. When eigenvalue deflation is used, the preconditioners turn out to be suitable for solving sequences of slowly changing systems or linear systems with different right-hand sides. Numerical results are provided, including the case of sequences arising in nonnegative linear least-squares problems solved by interior point methods.

Inversion of option prices for implied risk-neutral probability density functions: general theory and its applications to the natural gas market

This paper applies inverse theory to the estimation of the implied risk-neutral probability density function (PDF) from option prices. A general framework of inverting option prices for the implied risk-neutral PDF is formulated from the option pricing formula of Cox and Ross [J. Financial Econ., 1976, 3, 145–166]. To overcome the non-uniqueness and instability inherent in the option inverse problem, the smoothness requirement for the shape of the PDF and a prior model are introduced by a penalty function. Positivity constraints are included as a hard bond on the PDF values. The option inverse problem then becomes a non-negative least-squares problem that can be solved by classic methods such as the non-negative least-squares program of Lawson and Hanson [Solving Least Squares Problems, 1974 (Prentice-Hall: Englewood Cliffs, NJ)]. The best solution is not the one that gives the best fit to the observed option prices or provides the smoothest PDF, but the one that gives the optimal trade-off between the goodness-of-fit and smoothness of the estimated risk-natural PDF. The proposed inversion technique is compared with the models of Black–Scholes (BS), a mixture of two lognormals (MLN), Jarrow and Rudd's Edgeworth expansion (JR), and jump diffusion (JD) for the estimation of the PDF from the option prices associated with the September 2007 NYMEX natural gas futures. It is found that the inversion technique not only gives the best goodness-of-fit, but also a significantly better model resolution. An empirical study for the last three months of the September 2007 futures contract shows that the shapes of the estimated PDFs become more symmetric as the futures contract becomes closer to the expiration date. The dispersion of the estimated PDFs decreases with decreasing time to expiry, indicating the resolution of uncertainty with passing time.

An adaptive inverse scale space method for compressed sensing

In this paper we introduce a novel adaptive approach for solving `-minimization problems as frequently arising in compressed sensing, which is based on the recently introduced inverse scale space method. The scheme allows to efficiently compute minimizers by solving a sequence of low-dimensional nonnegative least-squares problems. We provide a detailed convergence analysis in a general setup as well as refined results under special conditions. In addition we discuss experimental observations in several numerical examples.

Computational experience with numerical methods for nonnegative least-squares problems

We discuss the solution of large-scale box-constrained linear least-squares problems by two recent affine-scaling methods: a cyclic Barzilai–Borwein strategy and an Inexact Newton-like method where a preconditioning technique allows for an efficient computation of the steps. A robust globally and fast locally convergent method based on the combination of the two procedures is presented along with extensive numerical results. Copyright © 2010 John Wiley & Sons, Ltd.