Approximate Message Passing Algorithm Research Articles

Effect of global shrinkage parameter of horseshoe prior in compressed sensing

This study investigates the effect of the global shrinkage parameter τ of a horseshoe prior, one of the global–local shrinkage priors, on linear regression in sparse signal processing. Statistical mechanics methods are employed to examine the accuracy of signal estimation. The phase diagram of the success and failure of signal recovery in noiseless compressed sensing with varying τ is discussed from the viewpoint of dynamic characterization of approximate message passing (AMP) as a solving algorithm and static characterization of the free-energy landscape. It is found that there exists a parameter region where the AMP algorithm can hardly recover the true signal, even though the true signal is locally stable. The analysis of the free-energy landscape also provides important insight into the optimal choice of τ.

On the atypical solutions of the symmetric binary perceptron

We study the random binary symmetric perceptron problem, focusing on the behavior of rare high-margin solutions. While most solutions are isolated, we demonstrate that these rare solutions are part of clusters of extensive entropy, heuristically corresponding to non-trivial fixed points of an approximate message-passing algorithm. We enumerate these clusters via a local entropy, defined as a Franz–Parisi potential, which we rigorously evaluate using the first and second moment methods in the limit of a small constraint density α (corresponding to vanishing margin κ ) under a certain assumption on the concentration of the entropy. This examination unveils several intriguing phenomena: (i) we demonstrate that these clusters have an entropic barrier in the sense that the entropy as a function of the distance from the reference high-margin solution is non-monotone when κ⩽1.429−α/log⁡α , while it is monotone otherwise, and that they have an energetic barrier in the sense that there are no solutions at an intermediate distance from the reference solution when κ⩽1.239−α/log⁡α . The critical scaling of the margin κ in −α/log⁡α corresponds to the one obtained from the earlier work of Gamarnik et al (2022 (arXiv:2203.15667)) for the overlap-gap property, a phenomenon known to present a barrier to certain efficient algorithms. (ii) We establish using the replica method that the complexity (the logarithm of the number of clusters of such solutions) versus entropy (the logarithm of the number of solutions in the clusters) curves are partly non-concave and correspond to very large values of the Parisi parameter, with the equilibrium being reached when the Parisi parameter diverges.

Approximate Message Passing for sparse matrices with application to the equilibria of large ecological Lotka–Volterra systems

This paper is divided into two parts. The first part is devoted to the study of a class of Approximate Message Passing (AMP) algorithms which are widely used in the fields of statistical physics, machine learning, or communication theory. The AMP algorithms studied in this part are those where the measurement matrix has independent elements, up to the symmetry constraint when this matrix is symmetric, with a variance profile that can be sparse. The AMP problem is solved by adapting the approach of Bayati, Lelarge, and Montanari (2015) to this matrix model.The Lotka–Volterra (LV) model is the standard model for studying the dynamical behavior of large dimensional ecological food chains. The second part of this paper is focused on the study of the statistical distribution of the globally stable equilibrium vector of a LV system in the situation where the random symmetric interaction matrix among the living species is sparse, and in the regime of large dimensions. This equilibrium vector is the solution of a Linear Complementarity Problem, which distribution is shown to be characterized through the AMP approach developed in the first part. In the large dimensional regime, this distribution is close to a mixture of a large number of truncated Gaussians.

Sparse Recovery of Finite Alphabet Signals Using AMP and Its Application in DoA Estimation

This study presents an approximate message passing (AMP)-based method for the reconstruction of sparse signals from multiple measurement vectors. Recently, Ziniel and Schniter have provided a general analysis of the AMP algorithm. They have assumed that the nonzero elements of the sparse signals have Gaussian distribution. This article extends their proposed method to the case that the nonzero elements of sparse vectors are chosen from a discrete finite alphabet. Therefore, the proposed algorithm is called approximate message passing with finite alphabet. Based on this assumption, the passing messages have been derived and the scheduling of the algorithm has been proposed using a three-step strategy. As an application, direction of arrival (DoA) estimation of digitally modulated signals is examined using simulations. The obtained results revealed that the proposed method would improve the DoA estimation accuracy compared to the conventional methods.

Fundamental limits in structured principal component analysis and how to reach them

How do statistical dependencies in measurement noise influence high-dimensional inference? To answer this, we study the paradigmatic spiked matrix model of principal components analysis (PCA), where a rank-one matrix is corrupted by additive noise. We go beyond the usual independence assumption on the noise entries, by drawing the noise from a low-order polynomial orthogonal matrix ensemble. The resulting noise correlations make the setting relevant for applications but analytically challenging. We provide characterization of the Bayes optimal limits of inference in this model. If the spike is rotation invariant, we show that standard spectral PCA is optimal. However, for more general priors, both PCA and the existing approximate message-passing algorithm (AMP) fall short of achieving the information-theoretic limits, which we compute using the replica method from statistical physics. We thus propose an AMP, inspired by the theory of adaptive Thouless-Anderson-Palmer equations, which is empirically observed to saturate the conjectured theoretical limit. This AMP comes with a rigorous state evolution analysis tracking its performance. Although we focus on specific noise distributions, our methodology can be generalized to a wide class of trace matrix ensembles at the cost of more involved expressions. Finally, despite the seemingly strong assumption of rotation-invariant noise, our theory empirically predicts algorithmic performance on real data, pointing at strong universality properties.

Variational Bayesian Inference Clustering-Based Joint User Activity and Data Detection for Grant-Free Random Access in mMTC

Tailor-made for massive connectivity and sporadic access, grant-free random access has become a promising candidate access protocol for massive machine-type communications (mMTC). Compared with conventional grant-based protocols, grant-free random access skips the exchange of scheduling information to reduce the signaling overhead, and facilitates sharing of access resources to enhance access efficiency. However, some challenges remain to be addressed in the receiver design, such as unknown identity of active users and multi-user interference (MUI) on shared access resources. In this work, we deal with the problem of joint user activity and data detection for grant-free random access. Specifically, the approximate message passing (AMP) algorithm is first employed to mitigate MUI and decouple the signals of different users. Then, we extend the data symbol alphabet to incorporate the null symbols from inactive users. In this way, the joint user activity and data detection problem is formulated as a clustering problem under the Gaussian mixture model. Furthermore, in conjunction with the AMP algorithm, a variational Bayesian inference based clustering (VBIC) algorithm is developed to solve this clustering problem. Simulation results show that, compared with state-of-art solutions, the proposed AMP-combined VBIC (AMP-VBIC) algorithm achieves a significant performance gain in detection accuracy.

Multi-Stage Training Optimization for Pilot Compression and Channel Estimation in Massive MIMO Systems Under Quasi-Sparse Channel Environment

The huge pilot overhead required for channel estimation is one of the key issues for massive multiple-input multiple-output (MIMO) systems. Existing compressed sensing and deep learning methods generally implement pilot compression under the assumption of the huge number of antennas and strict channel sparsity, while the channel estimation performance decreases sharply under quasi-sparse channel environment. Motivated by the powerful learning ability of deep learning for extracting the major characteristics of the target, we propose approximate message passing (AMP) algorithm-based multi-stage training optimization architecture to solve the above problem, where the whole system is considered as a deep neural network (DNN). In the first stage, the optimized sensing matrix of the AMP algorithm is learned by training to match the quasi-sparse channel. In the second stage, we optimize the linear coefficients and nonlinear shrinkage parameters of AMP to further improve the performance. In the third stage, all parameters are jointly optimized by training to make the whole system optimal. Numerical simulation results show that the proposed architecture can achieve high precision channel estimation under the premise of reducing pilot overhead effectively.

Approximate message passing with spectral initialization for generalized linear models* *This article is an updated version of: Mondelli M and Venkataramanan R 2021 Approximate message passing with spectral initialization for generalized linear models Proc. 24th Int. Conf. Artificial Intelligence and Statistics vol 130, ed A Banerjee and K Fukumizu pp 397–405.

We consider the problem of estimating a signal from measurements obtained via a generalized linear model. We focus on estimators based on approximate message passing (AMP), a family of iterative algorithms with many appealing features: the performance of AMP in the high-dimensional limit can be succinctly characterized under suitable model assumptions; AMP can also be tailored to the empirical distribution of the signal entries, and for a wide class of estimation problems, AMP is conjectured to be optimal among all polynomial-time algorithms. However, a major issue of AMP is that in many models (such as phase retrieval), it requires an initialization correlated with the ground-truth signal and independent from the measurement matrix. Assuming that such an initialization is available is typically not realistic. In this paper, we solve this problem by proposing an AMP algorithm initialized with a spectral estimator. With such an initialization, the standard AMP analysis fails since the spectral estimator depends in a complicated way on the design matrix. Our main contribution is a rigorous characterization of the performance of AMP with spectral initialization in the high-dimensional limit. The key technical idea is to define and analyze a two-phase artificial AMP algorithm that first produces the spectral estimator, and then closely approximates the iterates of the true AMP. We also provide numerical results that demonstrate the validity of the proposed approach.

Phase Transition of Total Variation Based on Approximate Message Passing Algorithm

In compressed sensing (CS), one seeks to down-sample some high-dimensional signals and recover them accurately by exploiting the sparsity of the signals. However, the traditional sparsity assumption cannot be directly satisfied in most practical applications. Fortunately, many signals-of-interest do at least exhibit a low-complexity representation with respect to a certain transformation. Particularly, total variation (TV) minimization is a notable example when the transformation operator is a difference matrix. Presently, many theoretical properties of total variation have not been completely explored, e.g., how to estimate the precise location of phase transitions and their rigorous understanding is still in its infancy. So far, the performance and robustness of the existing algorithm for phase transition prediction of TV model are not satisfactory. In this paper, we design a new approximate message passing algorithm to solve the above problems, called total variation vector approximate message passing (TV-VAMP) algorithm. To be specific, we first consider the problem from the Bayesian perspective, and formulate it as an optimization problem. Then, the vector factor graph for the TV model is constructed based on the formulized problem. Finally, the TV-VAMP algorithm is derived according to the idea of probabilistic inference in machine learning. Compared with the existing algorithm, our algorithm can be applied to a wider range of measurements distributions, including the non-zero-mean Gaussian distribution measurements matrix and ill-conditioned measurements matrix. Furthermore, in experiments with various settings, including different measurement distribution matrices, signal gradient sparsity, and measurement times, the proposed algorithm can almost reach the target mean squared error (−60 dB) with fewer iterations and better fit the empirical phase transition curve.

Iterative 2D sparse signal reconstruction with masked residual updates for automotive radar interference mitigation

Compressive sensing has attracted considerable attention in automotive radar interference mitigation. However, these algorithms usually cannot be applied directly to commercial automotive radar as most of them are computationally intense. In this paper, we therefore introduce a computationally efficient two-dimensional masked residual updates (2D MRU) compressive sensing framework. By utilizing the sparsity of the beat signal in the frequency domain, the range-Doppler (RD) spectrum can be reconstructed with the help of undistorted samples in the beat signal. Unlike the other schemes, where a 2D signal measurement is vectorized into a 1D signal, the proposed 2D MRU can directly take a 2D signal measurement and reconstruct the corresponding RD spectrum. Furthermore, the 2D MRU framework can be easily integrated into well-known optimization schemes such as basis pursuit, iterative hard thresholding, iterative soft thresholding, orthogonal matching pursuit, and approximate message-passing algorithm. In addition to the standard iterative thresholding algorithms, we propose a novel prior-model-based iterative thresholding method to further reduce the computation time and reconstruction error. Theoretical analysis shows that the proposed framework can successfully reconstruct the RD spectrum with high probability. Moreover, numerical experiments demonstrate the superiority of the proposed framework in terms of computational complexity.

Approximate Message Passing algorithms for rotationally invariant matrices

Approximate Message Passing (AMP) algorithms have seen widespread use across a variety of applications. However, the precise forms for their Onsager corrections and state evolutions depend on properties of the underlying random matrix ensemble, limiting the extent to which AMP algorithms derived for white noise may be applicable to data matrices that arise in practice. In this work, we study more general AMP algorithms for random matrices W that satisfy orthogonal rotational invariance in law, where W may have a spectral distribution that is different from the semicircle and Marcenko–Pastur laws characteristic of white noise. The Onsager corrections and state evolutions in these algorithms are defined by the free cumulants or rectangular free cumulants of the spectral distribution of W. Their forms were derived previously by Opper, Çakmak and Winther using nonrigorous dynamic functional theory techniques, and we provide rigorous proofs. Our motivating application is a Bayes-AMP algorithm for Principal Components Analysis, when there is prior structure for the principal components (PCs) and possibly nonwhite noise. For sufficiently large signal strengths and any non-Gaussian prior distributions for the PCs, we show that this algorithm provably achieves higher estimation accuracy than the sample PCs.

Approximate Message Passing Algorithms for Low Complexity OFDM-IM Detection

Low complexity approximate message passing (AMP) orthogonal frequency division multiplexing combined with index modulation (OFDM-IM) detection algorithms are proposed, which exploit the sparse structure of the frequency domain (FD) OFDM-IM symbols. To circumvent the high root mean square error (RMSE) in the conventional AMP algorithm, a minimum mean square error (MMSE) denoiser is proposed based on the classic Bayesian approach and on the state evolution of AMP. Our simulation results demonstrate that it is capable of improving both the RMSE as well as the convergence rate. However, in practice, the channel's diagonal FD matrix may be a non-Gaussian sensing matrix, hence a damping strategy is conceived. In conclusion, the proposed MMSE denoiser based damping-assisted AMP-aided detector strikes a compelling bit error ratio vs. complexity trade-off.

Cross Validation Aided Approximated Message Passing Algorithm for User Identification in mMTC

Recently, the approximate message passing (AMP) algorithm is advocated for overcoming the active user identification challenge in grant free (GF) massive random access (RA). However, requiring the perfect knowledge of user sparsity is a quite strong assumption of AMP algorithm in practical applications. Accordingly, we incorporate the cross validation (CV) method into the minimum mean square error (MMSE) denoiser based AMP algorithm, resulting a basic CV aided AMP algorithm (BCV-AMP) that can adaptively estimate the user sparsity. In order to further reduce its complexity, an improved CV aided MMSE denoiser based AMP algorithm (ICV-AMP) is proposed. Simulation results show that both the BCV-AMP and the ICV-AMP algorithms achieve almost the same user identification performance as the well-known MMSE denoiser based AMP algorithm, while the ICV-AMP algorithm has essentially the same complexity as that of the latter.

Estimation of low-rank matrices via approximate message passing

Consider the problem of estimating a low-rank matrix when its entries are perturbed by Gaussian noise, a setting that is also known as “spiked model” or “deformed random matrix.” If the empirical distribution of the entries of the spikes is known, optimal estimators that exploit this knowledge can substantially outperform simple spectral approaches. Recent work characterizes the asymptotic accuracy of Bayes-optimal estimators in the high-dimensional limit. In this paper, we present a practical algorithm that can achieve Bayes-optimal accuracy above the spectral threshold. A bold conjecture from statistical physics posits that no polynomial-time algorithm achieves optimal error below the same threshold (unless the best estimator is trivial). Our approach uses Approximate Message Passing (AMP) in conjunction with a spectral initialization. AMP algorithms have proved successful in a variety of statistical estimation tasks, and are amenable to exact asymptotic analysis via state evolution. Unfortunately, state evolution is uninformative when the algorithm is initialized near an unstable fixed point, as often happens in low-rank matrix estimation problems. We develop a new analysis of AMP that allows for spectral initializations, and builds on a decoupling between the outlier eigenvectors and the bulk in the spiked random matrix model. Our main theorem is general and applies beyond matrix estimation. However, we use it to derive detailed predictions for the problem of estimating a rank-one matrix in noise. Special cases of this problem are closely related—via universality arguments—to the network community detection problem for two asymmetric communities. For general rank-one models, we show that AMP can be used to construct confidence intervals and control false discovery rate. We provide illustrations of the general methodology by considering the cases of sparse low-rank matrices and of block-constant low-rank matrices with symmetric blocks (we refer to the latter as to the “Gaussian block model”).

Efficient hybrid Neumann series based MMSE assisted detection for 5G and beyond massive MIMO systems

Massive multiple-input multiple-output (M-MIMO) is one of the cutting edge technologies that provides significant improvement in throughput, coverage and spectral efficiency. The challenge with M-MIMO systems is to extract individual signals from the composite signal, thus making optimal detectors prohibitively complex. Recently, approximate message passing (AMP) and its variant detectors have gained substantial importance due to their decreased complexity and improved performance. However, AMP algorithm does not always converge. Non-linear detectors like vertical Bell-Labs layered space-iime improve the bit error rate (BER) but with high complexity, whereas, linear minimum mean square error (MMSE) detector offers low complexity while compromising on BER performance. Neumann series based MMSE detectors further reduce MMSE computational complexity, however, the BER performance remains the same. In this work, the authors propose a hybrid Neumann series based MMSE detector which decomposes the detected signal into its constituent components and apply a neighbourhood selection algorithm on the obtained components thus improving the overall performance. Another contribution of this work is derivation of an off-set value for optimised neighbourhood set selection that enables more accurate detection while further reducing computational complexity. Simulation results confirm that the proposed scheme outperforms aforementioned algorithms in terms of BER performance and computational complexity in a continuously changing Rayleigh channel.

Comparative Analysis of Data Detection Techniques for 5G Massive MIMO Systems

Massive multiple-input multiple-output (MIMO) is a backbone technology in the fifth-generation (5G) and beyond 5G (B5G) networks. It enhances performance gain, energy efficiency, and spectral efficiency. Unfortunately, a massive number of antennas need sophisticated processing to detect the transmitted signal. Although a detector based on the maximum likelihood (ML) is optimal, it incurs a high computational complexity, and hence, it is not hardware-friendly. In addition, the conventional linear detectors, such as the minimum mean square error (MMSE), include a matrix inversion, which causes a high computational complexity. As an alternative solution, approximate message passing (AMP) algorithm is proposed for data detection in massive MIMO uplink (UL) detectors. Although the AMP algorithm is converging extremely fast, the convergence is not guaranteed. A good initialization influences the convergence rate and affects the performance substantially together and the complexity. In this paper, we exploit several free-matrix-inversion methods, namely, the successive over-relaxation (SOR), the Gauss–Seidel (GS), and the Jacobi (JA), to initialize the AMP-based massive MIMO UL detector. In other words, hybrid detectors are proposed based on AMP, JA, SOR, and GS with an efficient initialization. Numerical results show that proposed detectors achieve a significant performance enhancement and a large reduction in the computational complexity.

Deep Learning for Beamspace Channel Estimation in Millimeter-Wave Massive MIMO Systems

Millimeter-wave massive multiple-input multiple-output (MIMO) can use a lens antenna array to considerably reduce the number of radio frequency (RF) chains, but channel estimation is challenging due to the number of RF chains is much smaller than that of antennas. By exploiting the sparsity of beamspace channels, the beamspace channel estimation can be formulated as a sparse signal recovery problem, which can be solved by the classical iterative algorithm named approximate message passing (AMP), and its corresponding version learned AMP (LAMP) realized by a deep neural network (DNN). However, these existing schemes cannot achieve satisfactory estimation accuracy. To improve the channel estimation performance, we propose a prior-aided Gaussian mixture LAMP (GM-LAMP) based beamspace channel estimation scheme in this paper. Specifically, based on the prior information that beamspace channel elements can be modeled by the Gaussian mixture distribution, we first derive a new shrinkage function to refine the AMP algorithm. Then, by replacing the original shrinkage function in the LAMP network with the derived Gaussian mixture shrinkage function, a prior-aided GM-LAMP network is developed to estimate the beamspace channel more accurately. Simulation results by using both the theoretical channel model and the ray-tracing based channel dataset show that, the proposed GM-LAMP network can achieve better channel estimation accuracy.

Speech enhancement based on approximate message passing

To overcome the limitations of conventional speech enhancement methods, such as inaccurate voice activity detector (VAD) and noise estimation, a novel speech enhancement algorithm based on the approximate message passing (AMP) is adopted. AMP exploits the difference between speech and noise sparsity to remove or mute the noise from the corrupted speech. The AMP algorithm is adopted to reconstruct the clean speech efficiently for speech enhancement. More specifically, the prior probability distribution of speech sparsity coefficient is characterized by Gaussian-model, and the hyper-parameters of the prior model are excellently learned by expectation maximization (EM) algorithm. We utilize the k-nearest neighbor (k-NN) algorithm to learn the sparsity with the fact that the speech coefficients between adjacent frames are correlated. In addition, computational simulations are used to validate the proposed algorithm, which achieves better speech enhancement performance than other four baseline methods-Wiener filtering, subspace pursuit (SP), distributed sparsity adaptive matching pursuit (DSAMP), and expectation-maximization Gaussian-model approximate message passing (EM-GAMP) under different compression ratios and a wide range of signal to noise ratios (SNRs).

RFF Based Detection for SCMA in Presence of PA Nonlinearity

The next-generation of communication systems must be capable of serving a significantly higher traffic generated by a large number of devices. To support massive connectivity over limited time-frequency resources, several non-orthogonal multiple access (NOMA) paradigms have emerged. In this context, sparse code multiple access (SCMA) based NOMA is particularly attractive due to its robustness to error-propagation, and its potentially high coding-gain with appropriate codebook design. However, the performance of SCMA based systems is severely degraded by distortions introduced by non-ideal hardware, e.g. power-amplifier (PA) nonlinearity. To mitigate such artefacts, in this letter, a random Fourier feature (RFF) based hybrid message passing algorithm (MPA) is proposed, and validated through computer simulations. Lastly, an analytical proof is presented that indicates that the use of RFFs significantly improves the convergence of the MPA in the presence of impairments, and renders error-rate performance equivalent to that of a linear Gaussian channel under approximate MPA.

Mutual Information and Optimality of Approximate Message-Passing in Random Linear Estimation

We consider the estimation of a signal from the knowledge of its noisy linear random Gaussian projections. A few examples where this problem is relevant are compressed sensing, sparse superposition codes, and code division multiple access. There has been a number of works considering the mutual information for this problem using the replica method from statistical physics. Here we put these considerations on a firm rigorous basis. First, we show, using a Guerra-Toninelli type interpolation, that the replica formula yields an upper bound to the exact mutual information. Secondly, for many relevant practical cases, we present a converse lower bound via a method that uses spatial coupling, state evolution analysis and the I-MMSE theorem. This yields a single letter formula for the mutual information and the minimal-mean-square error for random Gaussian linear estimation of all discrete bounded signals. In addition, we prove that the low complexity approximate message-passing algorithm is optimal outside of the so-called hard phase, in the sense that it asymptotically reaches the minimal-mean-square error. In this work spatial coupling is used primarily as a proof technique. However our results also prove two important features of spatially coupled noisy linear random Gaussian estimation. First there is no algorithmically hard phase. This means that for such systems approximate message-passing always reaches the minimal-mean-square error. Secondly, in a proper limit the mutual information associated to such systems is the same as the one of uncoupled linear random Gaussian estimation.