Underdetermined Linear System Research Articles

Sparse and nonnegative sparse D-MORPH regression

An underdetermined linear algebraic equation system \(\mathbf{y}={\varvec{\Phi }}\mathbf{x}\), where \({\varvec{\Phi }}\) is an \(m\times n (m<n)\) rectangular constant matrix with rank \(r\le m\) and \(\mathbf{y}\in \mathrm {Ran}({\varvec{\Phi }})\) (range of \({\varvec{\Phi }})\), has an infinite number of solutions. Diffeomorphic modulation under observable response preserving homotopy (D-MORPH) regression seeks a solution satisfying the extra requirement of minimizing a chosen cost function, \({\mathcal {K}}\). A wide variety of choices of the cost function makes it possible to achieve diverse goals, and hence D-MORPH regression has been successfully applied to solve a range of problems. In this paper, D-MORPH regression is extended to determine a sparse or a nonnegative sparse solution of the vector \(\mathbf{x}\). For this purpose, recursive reweighted least-squares (RRLS) minimization is adopted and modified to construct the cost function \({\mathcal {K}}\) for D-MORPH regression. The advantage of sparse and nonnegative sparse D-MORPH regression is that the matrix \({\varvec{\Phi }}\) does not need to have row-full rank, thereby enabling flexibility to search for sparse solutions \(\mathbf{x}\) with ancillary properties in practical applications. These tools are applied to (a) simulation data for quantum-control-mechanism identification utilizing high dimensional model representation (HDMR) modeling and (b) experimental mass spectral data for determining the composition of an unknown mixture of chemical species.

$NP/CMP$ Equivalence: A Phenomenon Hidden Among Sparsity Models $l_{0}$ Minimization and $l_{p}$ Minimization for Information Processing

In this paper, we have proved that in every underdetermined linear system $Ax=b$ , there corresponds a constant $p^{*}(A,b)>0$ such that every solution to the $l_{p}$ -norm minimization problem also solves the $l_{0}$ -norm minimization problem whenever $0 . This phenomenon is named $NP/CMP$ equivalence.

Nested Periodic Matrices and Dictionaries: New Signal Representations for Period Estimation

In this paper, we propose a new class of techniques to identify periodicities in data. We target the period estimation directly rather than inferring the period from the signal’s spectrum. By doing so, we obtain several advantages over the traditional spectrum estimation techniques such as DFT and MUSIC. Apart from estimating the unknown period of a signal, we search for finer periodic structure within the given signal. For instance, it might be possible that the given periodic signal was actually a sum of signals with much smaller periods. For example, adding signals with periods 3, 7, and 11 can give rise to a period 231 signal. We propose methods to identify these “hidden periods” 3, 7, and 11. We first propose a new family of square matrices called Nested Periodic Matrices (NPMs), having several useful properties in the context of periodicity. These include the DFT, Walsh–Hadamard, and Ramanujan periodicity transform matrices as examples. Based on these matrices, we develop high dimensional dictionary representations for periodic signals. Various optimization problems can be formulated to identify the periods of signals from such representations. We propose an approach based on finding the least $l_{2}$ norm solution to an under-determined linear system. Alternatively, the period identification problem can also be formulated as a sparse vector recovery problem and we show that by a slight modification to the usual $l_{1}$ norm minimization techniques, we can incorporate a number of new and computationally simple dictionaries.

Adaptive projected gradient thresholding methods for constrained l 0 problems

In this paper, we propose and analyze adaptive projected gradient thresholding (APGT) methods for finding sparse solutions of the underdetermined linear systems with equality and box constraints. The general convergence will be demonstrated, and in addition, the bound of the number of iterations is established in some special cases. Under suitable assumptions, it is proved that any accumulation point of the sequence generated by the APGT methods is a local minimizer of the underdetermined linear system. Moreover, the APGT methods, under certain conditions, can find all s-sparse solutions for accurate measurement cases and guarantee the stability and robustness for flawed measurement cases. Numerical examples are presented to show the accordance with theoretical results in compressed sensing and verify high out-of-sample performance in index tracking.

Load identification approach based on basis pursuit denoising algorithm

The information of the external loads is of great interest in many fields of structural analysis, such as structural health monitoring (SHM) systems or assessment of damage after extreme events. However, in most cases it is not possible to measure the external forces directly, so they need to be reconstructed. Load reconstruction refers to the problem of estimating an input to a dynamic system when the system output and the impulse response functions are usually the knowns. Generally, this leads to a so called ill-posed inverse problem, which involves solving an underdetermined linear system of equations. For most practical applications it can be assumed that the applied loads are not arbitrarily distributed in time and space, at least some specific characteristics about the external excitation are known a priori. In this contribution this knowledge was used to develop a more suitable force reconstruction method, which allows identifying the time history and the force location simultaneously by employing significantly fewer sensors compared to other reconstruction approaches. The properties of the external force are used to transform the ill-posed problem into a sparse recovery task. The sparse solution is acquired by solving a minimization problem known as basis pursuit denoising (BPDN). The possibility of reconstructing loads based on noisy structural measurement signals will be demonstrated by considering two frequently occurring loading conditions: harmonic excitation and impact events, separately and combined. First a simulation study of a simple plate structure is carried out and thereafter an experimental investigation of a real beam is performed.

Solving the Nonlinear Power Flow Problem Through General Solutions of Under-determined Linearised Systems

This work focuses on the solution of a set of algebraic nonlinear equations representing the steady state operation of electrical power systems. The classical modelling of the so-called power flow problem requires the statement of the power balance equations and the specification of some network variables. Usually Newton method is applied to solve these equations, which requires the solution of a linear system at each iteration. Here, the formulation of the power flow problem is modified by increasing the number of variables to be computed, such that an under-determined linear system is solved at each iteration. This strategy imparts flexibility to obtain solutions with respect to selected performance indexes. Numerical results obtained with test-systems ranging from 26 to 1916 equations and 30 to 2013 variables illustrate the main features of the proposed application.

Solving Basis Pursuit

The problem of finding a minimum ℓ 1 -norm solution to an underdetermined linear system is an important problem in compressed sensing, where it is also known as basis pursuit . We propose a heuristic optimality check as a general tool for ℓ 1 -minimization, which often allows for early termination by “guessing” a primal-dual optimal pair based on an approximate support. Moreover, we provide an extensive numerical comparison of various state-of-the-art ℓ 1 -solvers that have been proposed during the last decade, on a large test set with a variety of explicitly given matrices and several right-hand sides per matrix reflecting different levels of solution difficulty. The results, as well as improvements by the proposed heuristic optimality check, are analyzed in detail to provide an answer to the question which algorithm is the best.

Uniqueness Conditions for A Class of ℓ0-Minimization Problems

We consider a class of ℓ0-minimization problems, which is to search for the partial sparsest solution to an underdetermined linear system with additional constraints. We introduce several concepts, including lp-induced quasi-norm (0 &lt; p &lt; 1), maximal scaled spark and scaled mutual coherence, to develop several new uniqueness conditions for the partial sparsest solution to this class of ℓ0-minimization problems. A further improvement of some of these uniqueness criteria has been also achieved through the so-called concepts such as maximal scaled (sub)coherence rank.

The Sparsity of Underdetermined Linear System vialpMinimization for0&lt;p&lt;1

The sparsity problems have attracted a great deal of attention in recent years, which aim to find the sparsest solution of a representation or an equation. In the paper, we mainly study the sparsity of underdetermined linear system vialpminimization for0&lt;p&lt;1. We show, for a given underdetermined linear system of equationsAm×nX=b, that although it is not certain that the problem(Pp)(i.e.,minXXppsubject toAX=b, where0&lt;p&lt;1) generates sparser solutions as the value ofpdecreases and especially the problem(Pp)generates sparser solutions than the problem(P1)(i.e.,minXX1subject toAX=b), there exists a sparse constantγ(A,b)&gt;0such that the following conclusions hold whenp&lt;γ(A,b):(1)the problem(Pp)generates sparser solution as the value ofpdecreases;(2)the sparsest optimal solution to the problem(Pp)is unique under the sense of absolute value permutation;(3)letX1andX2be the sparsest optimal solution to the problems(Pp1)and(Pp2) (p1&lt;p2), respectively, and letX1not be the absolute value permutation ofX2. Then there existt1,t2∈[p1,p2]such thatX1is the sparsest optimal solution to the problem(Pt) (∀t∈[p1,t1])andX2is the sparsest optimal solution to the problem(Pt) (∀t∈(t2,p2]).

Signal Classification for Ground Penetrating Radar Using Sparse Kernel Feature Selection

This paper addresses the problem of feature selection for the classification of ground penetrating radar signals. We propose a new classification approach based on time-frequency analysis and sparse kernel feature selection. In the proposed approach, a time-frequency or a time-scale transform is first applied to the one-dimensional radar trace. Sparse kernel feature selection is then employed to extract an optimum set of features for classification. The sparse kernel method is formulated as an underdetermined linear system in a high-dimensional space, and the category labels of the training samples are used as measurements to select the most informative features. The proposed approach is evaluated through an industrial application of assessing railway ballast fouling conditions. Experimental results show that the proposed combination of sparse kernel feature selection and support vector machine classification yields very high classification rates using only a small number of features.

Positioning configurations with the lowest GDOP and their classification

The positioning configuration optimization is a basic problem in surveying, and the geometric dilution of precision (GDOP) is a key index to handle this problem. Simplex graphs as regular polygons and regular polyhedrons are the well-known configurations with the lowest GDOP. However, it has been proved that there are at most five kinds of regular polyhedrons. We analytically solve the GDOP minimization problem with arbitrary observational freedom to extend the current knowledge. The configuration optimization framework established is composed of the algebraic and geometric operators (including combination, reflection, collinear mapping, projection and three kinds of equivalence relations), basic properties to GDOP minimization (including continuity, combination invariant, reflection invariant, rotation invariant and collinear invariant) and the lowest GDOP configurations (including cones, regular polygons, regular polyhedrons, Descartes configuration, helical configuration and generalized Walker configuration, and their reflections and combinations). GDOP minimization criterion and D-maximization criterion both reduce to the same criterion matrices that the optimization becomes the problem for solving an underdetermined quadratic equation system. Making use of the concepts for solving underdetermined linear equation system, the concepts of base configuration (single classification) and general configuration (combined classification) are applied to the GDOP minimization to analytically solve the quadratic equation system. Firstly, the problems are divided into two subproblems by two kinds of GDOP to reveal the impact of the clock-offset on the configuration optimization, and it shows that the symmetry and uniformity play a key role in identifying the systematic errors. Then, the solution of the GDOP minimization is classified by the number of symmetry axes, that the base configurations with at least one symmetry axis and the general configurations without symmetry axis are categorized to be two large classifications. Complex configurations can be then generated by the combination and the reflection of those base configurations with simplex structure, and this indicates that completely solving the GDOP minimization needs to solve the simplex classifications primarily. Ultimately, constrained or unconstrained configuration optimization examples including GDOP distribution analysis, single-global satellite navigation system (GNSS) or multi-GNSS constellation design, configuration optimization of pseudolites and configuration design of buoys for underwater positioning are performed by employing the properties, lemmas, theorems and corollaries proposed.

Factored solution of nonlinear equation systems

This article generalizes a recently introduced procedure to solve nonlinear systems of equations, radically departing from the conventional Newton–Raphson scheme. The original nonlinear system is first unfolded into three simpler components: (i) an underdetermined linear system, (ii) a one-to-one nonlinear mapping with explicit inverse, and (iii) an overdetermined linear system. Then, instead of solving such an augmented system at once, a two-step procedure is proposed in which two equation systems are solved at each iteration, one of them involving the same symmetric matrix throughout the solution process. The resulting factored algorithm converges faster than Newton's method and, if carefully implemented, can be computationally competitive for large-scale systems. It can also be advantageous for dealing with infeasible cases.

A compressed sensing approach for efficient ensemble learning

This paper presents a method for improved ensemble learning, by treating the optimization of an ensemble of classifiers as a compressed sensing problem. Ensemble learning methods improve the performance of a learned predictor by integrating a weighted combination of multiple predictive models. Ideally, the number of models needed in the ensemble should be minimized, while optimizing the weights associated with each included model. We solve this problem by treating it as an example of the compressed sensing problem, in which a sparse solution must be reconstructed from an under-determined linear system. Compressed sensing techniques are then employed to find an ensemble which is both small and effective. An additional contribution of this paper, is to present a new performance evaluation method (a new pairwise diversity measurement) called the roulette-wheel kappa-error. This method takes into account the different weightings of the classifiers, and also reduces the total number of pairs of classifiers needed in the kappa-error diagram, by selecting pairs through a roulette-wheel selection method according to the weightings of the classifiers. This approach can greatly improve the clarity and informativeness of the kappa-error diagram, especially when the number of classifiers in the ensemble is large. We use 25 different public data sets to evaluate and compare the performance of compressed sensing ensembles using four different sparse reconstruction algorithms, combined with two different classifier learning algorithms and two different training data manipulation techniques. We also give the comparison experiments of our method against another five state-of-the-art pruning methods. These experiments show that our method produces comparable or better accuracy, while being significantly faster than the compared methods.

The Computational Complexity of the Restricted Isometry Property, the Nullspace Property, and Related Concepts in Compressed Sensing

This paper deals with the computational complexity of conditions which guarantee that the NP-hard problem of finding the sparsest solution to an underdetermined linear system can be solved by efficient algorithms. In the literature, several such conditions have been introduced. The most well-known ones are the mutual coherence, the restricted isometry property (RIP), and the nullspace property (NSP). While evaluating the mutual coherence of a given matrix is easy, it has been suspected for some time that evaluating RIP and NSP is computationally intractable in general. We confirm these conjectures by showing that for a given matrix A and positive integer k, computing the best constants for which the RIP or NSP hold is, in general, NP-hard. These results are based on the fact that determining the spark of a matrix is NP-hard, which is also established in this paper. Furthermore, we also give several complexity statements about problems related to the above concepts.

Fast linearized alternating direction method of multipliers for the augmented $$\ell _1$$ ℓ 1 -regularized problem

The problem of reconstructing a sparse signal from an underdetermined linear system captures many applications. And it has been shown that this NP-hard problem can be well approached via heuristically solving a convex relaxation problem where the $$\ell _1 $$ -norm is used to induce sparse structure. However, this convex relaxation problem is nonsmooth and thus not tractable in general, besides, this problem has a lower convergence rate. For these reasons, a signal reconstruction algorithm for solving the augmented $$\ell _1$$ -regularized problem is proposed in this paper. The separable structure of the new model enables us to solve the involved subproblems more efficiently by splitting the augmented Lagrangian function. Hence, an implementable numerical algorithm which called fast linearized alternating direction method of multipliers (FLADMM) is proposed to solve this novel model. Our experimental results show that the FLADMM method yields a higher peak signal-to-noise ratio reconstructed signal as well as a faster convergence rate at the same sampling rate as compared to the linearized Bregman method (LBM), the fast linearized Bregman iteration (FLBI) algorithm and the fast alternating direction method of multipliers (FADMM). Besides, this method is more robust than the LBM, FLBI and the FADMM algorithms at the same noise level.

A Primal Dual Active Set Algorithm With Continuation for Compressed Sensing

The success of compressed sensing relies essentially on the ability to efficiently find an approximately sparse solution to an under-determined linear system. In this paper, we developed an efficient algorithm for the sparsity promoting l <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> -regularized least squares problem by coupling the primal dual active set strategy with a continuation technique (on the regularization parameter). In the active set strategy, we first determine the active set from primal and dual variables, and then update the primal and dual variables by solving a low-dimensional least square problem on the active set, which makes the algorithm very efficient. The continuation technique globalizes the convergence of the algorithm, with provable global convergence under restricted isometry property (RIP). Further, we adopt two alternative methods, i.e., a modified discrepancy principle and a Bayesian information criterion, to choose the regularization parameter automatically. Numerical experiments indicate that our algorithm is very competitive with state-of-the-art algorithms in terms of accuracy and efficiency, without a priori information about the regularization parameter.

Recovering non-negative and combined sparse representations

The non-negative solution to an underdetermined linear system can be uniquely recovered sometimes, even without imposing any additional sparsity constraints. In this paper, we derive conditions under which a unique non-negative solution for such a system can exist, based on the theory of polytopes. Furthermore, we develop the paradigm of combined sparse representations, where only a part of the coefficient vector is constrained to be non-negative, and the rest is unconstrained (general). We analyze the recovery of the unique, sparsest solution, for combined representations, under three different cases of coefficient support knowledge: (a) the non-zero supports of non-negative and general coefficients are known, (b) the non-zero support of general coefficients alone is known, and (c) both the non-zero supports are unknown. For case (c), we propose the combined orthogonal matching pursuit algorithm for coefficient recovery and derive the deterministic sparsity threshold under which recovery of the unique, sparsest coefficient vector is possible. We quantify the order complexity of the algorithms, and examine their performance in exact and approximate recovery of coefficients under various conditions of noise. Furthermore, we also obtain their empirical phase transition characteristics. We show that the basis pursuit algorithm, with partial non-negative constraints, and the proposed greedy algorithm perform better in recovering the unique sparse representation when compared to their unconstrained counterparts. Finally, we demonstrate the utility of the proposed methods in recovering images corrupted by saturation noise.

Structural damage identification via a combination of blind feature extraction and sparse representation classification

This paper addresses two problems in structural damage identification: locating damage and assessing damage severity, which are incorporated into the classification framework based on the theory of sparse representation (SR) and compressed sensing (CS). The sparsity nature implied in the classification problem itself is exploited, establishing a sparse representation framework for damage identification. Specifically, the proposed method consists of two steps: feature extraction and classification. In the feature extraction step, the modal features of both the test structure and the reference structure model are first blindly extracted by the unsupervised complexity pursuit (CP) algorithm. Then in the classification step, expressing the test modal feature as a linear combination of the bases of the over-complete reference feature dictionary—constructed by concatenating all modal features of all candidate damage classes—builds a highly underdetermined linear system of equations with an underlying sparse representation, which can be correctly recovered by ℓ1-minimization; the non-zero entry in the recovered sparse representation directly assigns the damage class which the test structure (feature) belongs to. The two-step CP–SR damage identification method alleviates the training process required by traditional pattern recognition based methods. In addition, the reference feature dictionary can be of small size by formulating the issues of locating damage and assessing damage extent as a two-stage procedure and by taking advantage of the robustness of the SR framework. Numerical simulations and experimental study are conducted to verify the developed CP–SR method. The problems of identifying multiple damage, using limited sensors and partial features, and the performance under heavy noise and random excitation are investigated, and promising results are obtained.

An infeasible-point subgradient method using adaptive approximate projections

We propose a new subgradient method for the minimization of nonsmooth convex functions over a convex set. To speed up computations we use adaptive approximate projections only requiring to move within a certain distance of the exact projections (which decreases in the course of the algorithm). In particular, the iterates in our method can be infeasible throughout the whole procedure. Nevertheless, we provide conditions which ensure convergence to an optimal feasible point under suitable assumptions. One convergence result deals with step size sequences that are fixed a priori. Two other results handle dynamic Polyak-type step sizes depending on a lower or upper estimate of the optimal objective function value, respectively. Additionally, we briefly sketch two applications: Optimization with convex chance constraints, and finding the minimum ℓ 1-norm solution to an underdetermined linear system, an important problem in Compressed Sensing.

New and improved conditions for uniqueness of sparsest solutions of underdetermined linear systems

The uniqueness of sparsest solutions of underdetermined linear systems plays a fundamental role in compressed sensing theory. Several new algebraic concepts, including the sub-mutual coherence, scaled mutual coherence, coherence rank, and sub-coherence rank, are introduced in this paper in order to develop new and improved sufficient conditions for the uniqueness of sparsest solutions. The coherence rank of a matrix with normalized columns is the maximum number of absolute entries in a row of its Gram matrix that are equal to the mutual coherence. The main result of this paper claims that when the coherence rank of a matrix is low, the mutual-coherence-based uniqueness conditions for the sparsest solution of a linear system can be improved. Furthermore, we prove that the Babel-function-based uniqueness can be also improved by the so-called sub-Babel function. Moreover, we show that the scaled-coherence-based uniqueness conditions can be developed, and that the right-hand-side vector b of a linear system, the support overlap of solutions, and the range property of a transposed matrix can be also integrated into the criteria for the uniqueness of the sparsest solution of an underdetermined linear system.