Abstract
An algorithmic limit of compressed sensing or related variable-selection problems is analytically evaluated when a design matrix is given by an overcomplete random matrix. The replica method from statistical mechanics is employed to derive the result. The analysis is conducted through evaluation of the entropy, an exponential rate of the number of combinations of variables giving a specific value of fit error to given data which is assumed to be generated from a linear process using the design matrix. This yields the typical achievable limit of the fit error when solving a representative problem and includes the presence of unfavourable phase transitions preventing local search algorithms from reaching the minimum-error configuration. The associated phase diagrams are presented. A noteworthy outcome of the phase diagrams is that there exists a wide parameter region where any phase transition is absent from the high temperature to the lowest temperature at which the minimum-error configuration or the ground state is reached. This implies that certain local search algorithms can find the ground state with moderate computational costs in that region. Another noteworthy result is the presence of the random first-order transition in the strong noise case. The theoretical evaluation of the entropy is confirmed by extensive numerical methods using the exchange Monte Carlo and the multi-histogram methods. Another numerical test based on a metaheuristic optimisation algorithm called simulated annealing is conducted, which well supports the theoretical predictions on the local search algorithms. In the successful region with no phase transition, the computational cost of the simulated annealing to reach the ground state is estimated as the third order polynomial of the model dimensionality.
Highlights
Compressed sensing is a technique used to recover a high-dimensional signal from a limited number of measurements by utilising the fact that the signal of interest has redundancy and can be “sparse”; many of the coefficients are set to zero when described with an appropriate basis
There has been a surge of research of compressed sensing, which is based on a general idea that the signal has a sparse representation on an appropriate basis, because the idea can be shared in many other contexts such as data compression, multivariate regression, and variable selection
We developed an algorithm based on the so-called simulated annealing (SA) algorithm [25], which is a Monte Carlo (MC)-based optimisation solver, and demonstrated that it can efficiently find the minimum-error set for a wide range of parameters [26, 27]
Summary
Compressed sensing is a technique used to recover a high-dimensional signal from a limited number of measurements by utilising the fact that the signal of interest has redundancy and can be “sparse”; many of the coefficients are set to zero when described with an appropriate basis. In the context of compressed sensing, the design matrix A represents the measurement process, and given A and y, we try to infer x0 for the situation M < N This is an underdetermined problem and the sparsity assumption that the number of nonzero components of x0 is smaller than M is needed for solving it. It may be better to focus less on the signal sources and rely more on the less informative prior According to this idea, in the context of data compression, the present authors recently proposed a variable selection criterion based on the following widely-used optimisation formulation [24]:. The MAP estimator with the uninformative flat prior φ(x) = 1 allows us to bypass this problem and can yield better performance than the Bayes estimator (19) in certain mismatching cases These reasons naturally motivate us to investigate eq (15) instead of eq (19). In the following discussions, any model selection using this criterion is not performed and the entire treatment in the main text does not inherit any of these problems
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callMore From: Journal of Statistical Mechanics: Theory and Experiment
Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
