Basis Pursuit Problem Research Articles

The sparse Kaczmarz method with surrogate hyperplane for the regularized basis pursuit problem

The Sparse Kaczmarz method is a famous and widely used iterative method for solving the regularized basis pursuit problem. A general scheme of the surrogate hyperplane sparse Kaczmarz method is proposed. In particular, a class of residual-based surrogate hyperplane sparse Kaczmarz method is introduced and the implementations are well discussed. Their convergence theories are proved and the linear convergence rates are studied and compared in details. Numerical experiments verify the efficiency of the proposed methods.

A randomized sparse Kaczmarz solver for sparse signal recovery via minimax‐concave penalty

The randomized sparse Kaczmarz (RSK) method is an algorithm used to calculate sparse solutions for the basis pursuit problem. In this paper, we propose an algorithm framework for computing sparse solutions of linear systems, which includes the sparse Kaczmarz and sparse block Kaczmarz algorithms. In order to overcome the limitations of the penalty, we design an effective and new randomized sparse Kaczmarz algorithm (RSK‐MCP) based on the non‐convex minimax‐concave penalty (MCP) in sparse signal reconstruction. Additionally, we prove that the RSK‐MCP algorithm is equivalent to the randomized coordinate descent method for the corresponding dual problem. Based on this result, we demonstrate that the RSK‐MCP algorithm exhibits linear convergence, meaning it converges to a sparse solution of the MCP model when the regularization of MCP is a strongly convex function. Numerical experiments indicate that the RSK‐MCP algorithm outperforms RSK‐L1 in terms of both efficiency and accuracy.

The ADMM algorithm for audio signal recovery and performance modification with the dual Douglas-Rachford dynamical system

<abstract><p>Practitioners employ operator splitting methods—such as alternating direction method of multipliers (ADMM) and its "dual" Douglas-Rachford method (DR)—to solve many kinds of optimization problems. We provide a gentle introduction to these algorithms, and illustrations of their duality-like relationship in the context of solving basis pursuit problems for audio signal recovery. Recently, researchers have used the dynamical systems associated with the iterates of splitting methods to motivate the development of schemes to improve performance. These developments include a class of methods that act by iteratively minimizing surrogates for a Lyapunov function for the dynamical system. An exemplar of this class is currently state-of-the-art for the feasibility problem of finding wavelets with special structure. Early experimental evidence has also suggested that, when implemented in a primal-dual (ADMM and DR) framework, this exemplar may provide improved performance for basis pursuit problems. We provide a reasonable way to compute the updates for this exemplar, and we study the application of this method to the aforementioned basis pursuit audio problems. We provide experimental results and visualizations of the dynamical system for the dual DR sequence. We observe that for highly structured problems with real data, the algorithmic behavior is noticeably different than for randomly generated problems.</p></abstract>

Extended randomized Kaczmarz method for sparse least squares and impulsive noise problems

The Extended Randomized Kaczmarz method is a well known iterative scheme which can find the Moore-Penrose inverse solution of a possibly inconsistent linear system and requires only one additional column of the system matrix in each iteration in comparison with the standard randomized Kaczmarz method. Also, the Sparse Randomized Kaczmarz method has been shown to converge linearly to a sparse solution of a consistent linear system. Here, we combine both ideas and propose an Extended Sparse Randomized Kaczmarz method. We show linear expected convergence to a sparse least squares solution in the sense that an extended variant of the regularized basis pursuit problem is solved. Moreover, we generalize the additional step in the method and prove convergence to a more abstract optimization problem. We demonstrate numerically that our method can find sparse least squares solutions of real and complex systems if the noise is concentrated in the complement of the range of the system matrix and that our generalization can handle impulsive noise.

Matrix-Form Neural Networks for Complex-Variable Basis Pursuit Problem With Application to Sparse Signal Reconstruction.

In this article, a continuous-time complex-valued projection neural network (CCPNN) in a matrix state space is first proposed for a general complex-variable basis pursuit problem. The proposed CCPNN is proved to be stable in the sense of Lyapunov and to be globally convergent to the optimal solution under the condition that the sensing matrix is not row full rank. Furthermore, an improved discrete-time complex projection neural network (IDCPNN) is proposed by discretizing the CCPNN model. The proposed IDCPNN consists of a two-step stop strategy to reduce the calculational cost. The proposed IDCPNN is theoretically guaranteed to be global convergent to the optimal solution. Finally, the proposed IDCPNN is applied to the reconstruction of sparse signals based on compressed sensing. Computed results show that the proposed IDCPNN is superior to related complex-valued neural networks and conventional basis pursuit algorithms in terms of solution quality and computation time.

The MMV tail null space property and DOA estimations by tail-[formula omitted] minimization

The tail-minimization approach is applied to the joint sparse multiple measurement vector (MMV) model and direction of arrival (DOA) estimations. The mechanism is to estimate the support T of the jointly sparse matrix and minimize the energy in the complement Tc of T during each iteration. An MMV tail-null-space property (MMV-tail-NSP) is derived, which is shown to be necessary and sufficient for the unique solution of the MMV tail minimization problem. The MMV-tail-NSP condition is also shown to be more likely to hold than that of the conventional MMV-NSP for any given MMV basis pursuit problem. Two recovery guarantees and error bound analyses are also derived based on the MMV-robust-tail-NSP condition and merely the MMV-tail-NSP assumption, respectively. Study shows that the MMV tail-minimization approach is among the most effective techniques for the MMV DOA model. In particular, the MMV tail-minimization approach is able to recover signals of rank-enriched sparsity level K0≡[spark(A)−1+rank(X)]/2. In fact, this approach can handle signal recovery of greater sparsity levels than that by others. The advantages of the MMV tail-minimization approach are also reflected in numerous aspects such as handling low signal-to-noise ratio (SNR), limited number of snapshots, and correlated source signals. Results are more accurate with higher resolution as well. It is also capable of detecting the number of sources and estimating DOAs simultaneously. All these characteristics and advantages are fully evidenced by extensive simulation studies.

An Extended Reweighted ℓ1 Minimization Algorithm for Image Restoration

This paper proposes an effective extended reweighted ℓ1 minimization algorithm (ERMA) to solve the basis pursuit problem minu∈Rn{||u||1:Au=f} in compressed sensing, where A∈Rm×n, m≪n. The fast algorithm is based on linearized Bregman iteration with soft thresholding operator and generalized inverse iteration. At the same time, it also combines the iterative reweighted strategy that is used to solve minu∈Rn{||u||pp:Au=f} problem, with the weight ωi(u,p)=(ε+|ui|2)p/2−1. Numerical experiments show that this ℓ1 minimization persistently performs better than other methods. Especially when p=0, the restored signal by the algorithm has the highest signal to noise ratio. Additionally, this approach has no effect on workload or calculation time when matrix A is ill-conditioned.

A Laplacian approach to ell _1-norm minimization

We propose a novel differentiable reformulation of the linearly-constrained ell _1 minimization problem, also known as the basis pursuit problem. The reformulation is inspired by the Laplacian paradigm of network theory and leads to a new family of gradient-based methods for the solution of ell _1 minimization problems. We analyze the iteration complexity of a natural solution approach to the reformulation, based on a multiplicative weights update scheme, as well as the iteration complexity of an accelerated gradient scheme. The results can be seen as bounds on the complexity of iteratively reweighted least squares (IRLS) type methods of basis pursuit.

Многоблочный метод ADMM с ускорением Нестерова

Метод ADMM (alternating direction methods of multipliers) широко используется для решения многих оптимизационных задач с помощью параллельных вычислений. Этот метод особенно важен при решении задач, возникающих в машинном обучении, математической статистике, распознавании образов, восстановлении сигналов, обработке данных значительного объема. ADMM позволяет решать оптимизационные задачи, целевая функция которых представляет собой сумму гладкой и негладкой функций, что характерно для задач со штрафной функцией. Целью этой статьи является разработка улучшенной версии ADMM с лучшей скоростью сходимости, чем существующие методы. В статье описаны существующие подходы для улучшения эффективности ADMM-метода, приведены основные работы по данной тематике и предложен новый метод, базирующийся на комбинации двух уже существующих подходов - разбиение начальной оптимизационной задачи на N подзадач и применение многоблочного подхода для разрешение и вычисление ускорения Нестерова на каждой итерации. Приведены теоретическое обоснование сходимости данного метода и установлены условия сходимости. Реализован предлагаемый алгоритм языка программирования Python и применен для решения задачи обмена с генерируемыми случайным образом данными, задачи поиска базиса и задачи LASSO с ограничениями. Приведены результаты сравнения эффективности многоблочного ADMM с ускорением Нестерова и существующих многоблочного и стандартного двухблочного ADMM. Многоблочный ADMM с ускорением Нестерова показал лучшую вычислительную эффективность, чем уже существующие методы. Еще одним преимуществом предлагаемого метода является его удобство для проведения параллельных вычислений с применением современных многопроцессорных систем. В связи с большими объемами данных, обработка которых требует значительного времени при решении оптимизационных задач, предложенный метод имеет важное практическое значение, поскольку он значительно превышает по скорости известные аналоги. Использование предложенного метода позволит решить практически важные задачи большого объема, применив параллельные вычисления.

AFISTA: Accelerated FISTA for sparse signal recovery and compressive sensing

This paper presents a new fast iterative shrinkage-thresholding algorithm, termed AFISTA. The essential idea is to improve the convergence rate of FISTA using a new continuation strategy leading to a less number of iterations compared to FISTA. The convergence theorem of the AFISTA is proposed. In order to further accelerate the AFISTA method, it is equipped with the Barzilai-Borwein (BB) method. Also, for applications with orthogonal sensing matrix A, we proposed a specialized version of the AFISTA method. AFISTA is tailored for solving the basis pursuit problem which can be applied successfully on a variety of problems arising in signal and image processing issues such as sparse signal recovery, signal and image denoising, image restoration, and compressive sensing. To show the efficiency of the method, we compare our results with generalizations of linearized Bregman and fixed - point continuation (FPC) methods in sparse signal recovery applications, with split Bregman method in compressive sensing for sparse MRI and with Gradient projection for sparse reconstruction (GPSR) method in image deconvolution. Numerical results demonstrate that AFISTA overcomes all of the compared methods in convergence rate and some of them in both convergence rate and quality of reconstructed results.

Functional penalised basis pursuit on spheres

In this paper, we propose a unified theoretical and practical spherical approximation framework for functional inverse problems on the hypersphere. More specifically, we consider recovering spherical fields directly in the continuous domain using functional penalised basis pursuit problems with gTV regularisation terms. Our framework is compatible with various measurement types as well as non-differentiable convex cost functionals. Via a novel representer theorem, we characterise their solution sets in terms of spherical splines with sparse innovations. We use this result to derive an approximate canonical spline-based discretisation scheme, with vanishing approximation error. To solve the resulting finite-dimensional optimisation problem, we propose an efficient and provably convergent primal-dual splitting algorithm. We illustrate the versatility of our framework on real-life examples from the field of environmental sciences.

Convergence of Slice-Based Block Coordinate Descent Algorithm for Convolutional Sparse Coding

Convolutional sparse coding (CSC) models are becoming increasingly popular in the signal and image processing communities in recent years. Several research studies have addressed the basis pursuit (BP) problem of the CSC model, including the recently proposed local block coordinate descent (LoBCoD) algorithm. This algorithm adopts slice-based local processing ideas and splits the global sparse vector into local vector needles that are locally computed in the original domain to obtain the encoding. However, a convergence theorem for the LoBCoD algorithm has not been given previously. This paper presents a convergence theorem for the LoBCoD algorithm which proves that the LoBCoD algorithm will converge to its global optimum at a rate of O1/k. A slice-based multilayer local block coordinate descent (ML-LoBCoD) algorithm is proposed which is motivated by the multilayer basis pursuit (ML-BP) problem and the LoBCoD algorithm. We prove that the ML-LoBCoD algorithm is guaranteed to converge to the optimal solution at a rate O1/k. Preliminary numerical experiments demonstrate the better performance of the proposed ML-LoBCoD algorithm compared to the LoBCoD algorithm for the BP problem, and the loss function value is also lower for ML-LoBCoD than LoBCoD.

An improved algorithm for basis pursuit problem and its applications

We propose an algorithm for solving the basis pursuit problem minu∈Cn{∥u∥1:Au=f}. Our starting motivation is the algorithm for compressed sensing, proposed by Qiao, Li and Wu, which is based on linearized Bregman iteration with generalized inverse. Qiao, Li and Wu defined new algorithm for solving the basis pursuit problem in compressive sensing using a linearized Bregman iteration and the iterative formula of linear convergence for computing the matrix generalized inverse. In our proposed approach, we combine a partial application of the Newton’s second order iterative scheme for computing the generalized inverse with the Bregman iteration. Our scheme takes lesser computational time and gives more accurate results in most cases. The effectiveness of the proposed scheme is illustrated in two applications: signal recovery from noisy data and image deblurring.

On Multi-Layer Basis Pursuit, Efficient Algorithms and Convolutional Neural Networks.

Parsimonious representations are ubiquitous in modeling and processing information. Motivated by the recent Multi-Layer Convolutional Sparse Coding (ML-CSC) model, we herein generalize the traditional Basis Pursuit problem to a multi-layer setting, introducing similar sparse enforcing penalties at different representation layers in a symbiotic relation between synthesis and analysis sparse priors. We explore different iterative methods to solve this new problem in practice, and we propose a new Multi-Layer Iterative Soft Thresholding Algorithm (ML-ISTA), as well as a fast version (ML-FISTA). We show that these nested first order algorithms converge, in the sense that the function value of near-fixed points can get arbitrarily close to the solution of the original problem. We further show how these algorithms effectively implement particular recurrent convolutional neural networks (CNNs) that generalize feed-forward ones without introducing any parameters. We present and analyze different architectures resulting from unfolding the iterations of the proposed pursuit algorithms, including a new Learned ML-ISTA, providing a principled way to construct deep recurrent CNNs. Unlike other similar constructions, these architectures unfold a global pursuit holistically for the entire network. We demonstrate the emerging constructions in a supervised learning setting, consistently improving the performance of classical CNNs while maintaining the number of parameters constant.

Compressive sensing in electrical impedance tomography for breathing monitoring

Objective: Electrical impedance tomography (EIT) is a functional imaging technique in which cross-sectional images of structures are reconstructed based on boundary trans-impedance measurements. Continuous functional thorax monitoring using EIT has been extensively researched. Increasing the number of electrodes, number of planes and frame rate may improve clinical decision making. Thus, a limiting factor in high temporal resolution, 3D and fast EIT is the handling of the volume of raw impedance data produced for transmission and its subsequent storage. Owing to the periodicity (i.e. sparsity in frequency domain) of breathing and other physiological variations that may be reflected in EIT boundary measurements, data dimensionality may be reduced efficiently at the time of sampling using compressed sensing techniques. This way, a fewer number of samples may be taken. Approach: Measurements using a 32-electrode, 48-frames-per-second EIT system from 30 neonates were post-processed to simulate random demodulation acquisition method on 2000 frames (each consisting of 544 measurements) for compression ratios (CRs) ranging from 2 to 100. Sparse reconstruction was performed by solving the basis pursuit problem using SPGL1 package. The global impedance data (i.e. sum of all 544 measurements in each frame) was used in the subsequent studies. The signal to noise ratio (SNR) for the entire frequency band (0 Hz–24 Hz) and three local frequency bands were analysed. A breath detection algorithm was applied to traces and the subsequent error-rates were calculated while considering the outcome of the algorithm applied to a down-sampled and linearly interpolated version of the traces as the baseline. Main results: SNR degradation was generally proportional with CR. The mean degradation for 0 Hz–8 Hz (of interest for the target physiological variations) was below ~15 dB for all CRs. The error-rates in the outcome of the breath detection algorithm in the case of decompressed traces were lower than those associated with the corresponding down-sampled traces for CR ⩾ 25, corresponding to sub-Nyquist rate for breathing frequency. For instance, the mean error-rate associated with CR = 50 was ~60% lower than that of the corresponding down-sampled traces. Significance: To the best of our knowledge, no other study has evaluated the applicability of compressive sensing techniques on raw boundary impedance data in EIT. While further research should be directed at optimising the acquisition and decompression techniques for this application, this contribution serves as the baseline for future efforts.

Data-driven compressive sensing and applications in uncertainty quantification

In this paper, we propose a data-driven compressive sensing method for the effective and sparse representation of solutions to a stochastic system, in which a problem-dependent basis is constructed and used to exploit the sparse patterns. Given what appears to be a highly incomplete set of sample observations, the essential ingredients of the proposed method involve (i) estimating the covariance function from low-fidelity simulations, (ii) extracting the most dominant energetic modes from the Karhunen-Loève analysis, and (iii) solving the basis pursuit problem associated with data-driven functions for high-fidelity models. Compared with other conventional compressive sensing methods that use a problem-independent basis, our approach can significantly increase the sparsity of expansion coefficients in many situations. When applied to uncertainty quantification problems such as partial differential equations with random input data, numerical experiments are carried out to show the effectiveness of our data-driven method in recovering the sparse solution globally over the physical domain.

Sparsest representations and approximations of an underdetermined linear system

In an underdetermined linear system of equations, constrained l1 minimization methods such as the basis pursuit or the lasso are often used to recover one of the sparsest representations or approximations of the system. The null space property is a sufficient and ‘almost’ necessary condition to recover a sparsest representation with the basis pursuit. Unfortunately, this property cannot be easily checked. On the other hand, the mutual coherence is an easily checkable sufficient condition insuring the basis pursuit to recover one of the sparsest representations. Because the mutual coherence condition is too strong, it is hardly met in practice. Even if one of these conditions holds, to our knowledge, there is no theoretical result insuring that the lasso solution is one of the sparsest approximations. In this article, we study a novel constrained problem that gives, without any condition, one of the sparsest representations or approximations. To solve this problem, we provide a numerical method and we prove its convergence. Numerical experiments show that this approach gives better results than both the basis pursuit problem and the reweighted l1 minimization problem.

Performance Analysis and Simulation of Numerical Algorithm for Quadratic Basis Pursuit Problem

Signal reconstruction is an important technique for compressive sensing problem. At present, when the relationship between the measurements and the unknowns is non-linear, the quadratic basis pursuit (QBP) is a practical signal reconstruction algorithm to solve the problem. However, it is computationally demanding. In this paper, a modified QBP method based on alternating direction method of multipliers (ADMM) is proposed. The performance of this algorithm is investigated by numerical analysis. Compared with traditional convex optimization tool, it has smaller computational time.

Seismic data interpolation using generalised velocity‐dependent seislet transform

ABSTRACTData interpolation is an important step for seismic data analysis because many processing tasks, such as multiple attenuation and migration, are based on regularly sampled seismic data. Failed interpolations may introduce artifacts and eventually lead to inaccurate final processing results. In this paper, we generalised seismic data interpolation as a basis pursuit problem and proposed an iteration framework for recovering missing data. The method is based on non‐linear iteration and sparse transform. A modified Bregman iteration is used for solving the constrained minimisation problem based on compressed sensing. The new iterative strategy guarantees fast convergence by using a fixed threshold value. We also propose a generalised velocity‐dependent formulation of the seislet transform as an effective sparse transform, in which the non‐hyperbolic normal moveout equation serves as a bridge between local slope patterns and moveout parametres in the common‐midpoint domain. It can also be reduced to the traditional velocity‐dependent seislet if special heterogeneity parametre is selected. The generalised velocity‐dependent seislet transform predicts prestack reflection data in offset coordinates, which provides a high compression of reflection events. The method was applied to synthetic and field data examples, and the results show that the generalised velocity‐dependent seislet transform can reconstruct missing data with the help of the modified Bregman iteration even for non‐hyperbolic reflections under complex conditions, such as vertical transverse isotropic (VTI) media or aliasing.

Sparse Approximate Reconstruction Decomposed by Two Optimization Problems

We propose a novel sparse signal reconstruction method aiming to directly minimize $$\ell _{0}$$ -quasinorm. Based on the smoothed $$\ell _{0}$$ -quasinorm, we show that there exists an unconstrained optimization problem such that both this problem and the basis pursuit problem are subproblems of the $$\ell _{0}$$ minimization problem. Moreover, we can obtain a sparse solution to the $$\ell _{0}$$ minimization by solving these two subproblems. In addition, we establish the relation between solutions to the $$\ell _{0}$$ minimization and the least square solutions of a linear system. Finally, we present some numerical experiments to illustrate our results.