Generalized Belief Propagation Research Articles

Generalized Belief Propagation Algorithms for Decoding of Surface Codes

Belief propagation (BP) is well-known as a low complexity decoding algorithm with a strong performance for important classes of quantum error correcting codes, e.g. notably for the quantum low-density parity check (LDPC) code class of random expander codes. However, it is also well-known that the performance of BP breaks down when facing topological codes such as the surface code, where naive BP fails entirely to reach a below-threshold regime, i.e. the regime where error correction becomes useful. Previous works have shown, that this can be remedied by resorting to post-processing decoders outside the framework of BP. In this work, we present a generalized belief propagation method with an outer re-initialization loop that successfully decodes surface codes, i.e. opposed to naive BP it recovers the sub-threshold regime known from decoders tailored to the surface code and from statistical-mechanical mappings. We report a threshold of 17% under independent bit-and phase-flip data noise (to be compared to the ideal threshold of 20.6%) and a threshold value of 14% under depolarizing data noise (compared to the ideal threshold of 18.9%), which are on par with thresholds achieved by non-BP post-processing methods.

Performing Bayesian Network Inference Using Amortized Region Approximation with Graph Factorization

Exact inference for large, directed graphical models, also known as Bayesian networks (BNs), can be intractable as the space complexity grows exponentially in the tree-width of the model. Approximate inference, such as generalized belief propagation (GBP), is used instead. GBP treats inference as the Bethe/Kikuchi energy function optimization problem. The solution is found using iterative message passing, which is inefficient and convergent problematic. Recent progress on amortized technique for GBP is an attractive alternative solution that can optimize the Bethe/Kikuchi energy function using (deep) neural networks, requiring no message passing. Despite being efficient, the amortized technique for GBP is applied to undirected graphical models with specific structures and factors, with no guarantee of the approximation quality. This is because the energy function to be amortized is defined by a region (or factor) graph that is ad hoc and difficult to construct to ensure sensible approximations. This paper proposes a new amortized GBP algorithm applied to BN for efficient inference. The proposed algorithm is composed of the following: (i) a new pairwise conversion (PWC) algorithm that converts all the conditional probability distributions in the BN into pairwise factors to facilitate efficient region graph constructions; (ii) following PWC, an improved loop structured region graph (LSRG) algorithm was derived to generate a valid region graph satisfying desired regional properties; and (iii) the energy function defined by the proposed PWC-LSRG region graph can be directly amortized using (deep) neural networks to ensure sensible approximations. Empirical studies show that the proposed amortized PWC-LSRG algorithm is of practical use and significantly improves convergence and efficiency compared to conventional algorithms.

Gaussian Belief Propagation Solvers for Nonsymmetric Systems of Linear Equations

In this paper, we argue for the utility of deterministic inference in the classical problem of numerical linear algebra, that of solving a linear system. We show how the Gaussian belief propagation solver, known to work for symmetric matrices can be modified to handle nonsymmetric matrices. Furthermore, we introduce a new algorithm for matrix inversion that corresponds to the generalized belief propagation derived from the cluster variation method (or Kikuchi approximation). We relate these algorithms to LU and block LU decompositions and provide certain guarantees based on theorems from the theory of belief propagation. All proposed algorithms are compared with classical solvers (e.g., Gauss-Seidel, BiCGSTAB) with application to linear elliptic equations. We also show how the Gaussian belief propagation can be used as multigrid smoother, resulting in a substantially more robust solver than the one based on the Gauss-Seidel iterative method.

Multi-Layer Bilinear Generalized Approximate Message Passing

In this paper, we extend the bilinear generalized approximate message passing (BiG-AMP) approach, originally proposed for high-dimensional generalized bilinear regression, to the multi-layer case for the handling of cascaded problem such as matrix-factorization problem arising in relay communication among others. Assuming statistically independent matrix entries with known priors, the new algorithm called ML-BiGAMP could approximate the general sum-product loopy belief propagation (LBP) in the high-dimensional limit enjoying a substantial reduction in computational complexity. We demonstrate that, in large system limit, the asymptotic MSE performance of ML-BiGAMP could be fully characterized via a set of simple one-dimensional equations termed state evolution (SE). We establish that the asymptotic MSE predicted by ML-BiGAMP' SE matches perfectly the exact MMSE predicted by the replica method, which is well-known to be Bayes-optimal but infeasible in practice. This consistency indicates that the ML-BiGAMP may still retain the same Bayes-optimal performance as the MMSE estimator in high-dimensional applications, although ML-BiGAMP's computational burden is far lower. As an illustrative example of the general ML-BiGAMP, we provide a detector design that could estimate the channel fading and the data symbols jointly with high precision for the two-hop amplify-and-forward relay communication systems.

Collaborative Multiple UAVs Navigation With GPS/INS/UWB Jammers Using Sigma Point Belief Propagation

Location awareness and navigation promote varieties of emerging applications of mobile collaborative multiple uncrewed aerial vehicles (UAVs). Cooperative UAVs fuse the global position system (GPS), inertial navigation systems (INS), peer to peer ranging radios derived from relative navigation of ultra-wideband (UWB) under complicated environments. Those information sources can be incorporated into spatiotemporal cooperation posited by the intra-user measurement of INS and GPS, and the inter-user measurement of the relative navigation of swarm UAVs. This paper considers the localization and navigation of multiple collaborative UAVs in networks with GPS/INS/UWB jammers in the case that the measurements are missed or randomly delayed by a sampling period. In a navigation situation with a partially denied navigation signals (e.g.GPS Jammers for some UAVs, UWB jammers for others, etc.), we propose an improved method of cooperation location for the swarm, allowing measurement jammers concerning the normal sigma point belief propagation (SPBP). This algorithm integrates message passing based on the Bayesian framework, a sigma point belief propagation of random packet loss (SPBP-RPL) to exploit spatiotemporal cooperation and measurement knowledge. Compared with existing general sigma point belief propagation, the advantages of the novel method are validated through a simulation of swarm UAVs with GPS/INS/UWB. Results show that the algorithm of combining spatiotemporal cooperation with measurement knowledge reduces the location uncertainty of swarm UAVs agents and improves location accuracy remarkably.

A Deliberate Bit Flipping Coding Scheme for Data-Dependent Two-Dimensional Channels

In this paper, we present a deliberate bit flipping (DBF) coding scheme for binary two-dimensional (2-D) channels, where specific patterns in channel inputs are the significant cause of errors. The idea is to eliminate a constrained encoder and, instead, embed a constraint into an error correction codeword that is arranged into a 2-D array by deliberately flipping the bits that violate the constraint. The DBF method relies on the error correction capability of the code being used so that it should be able to correct both deliberate errors and channel errors. Therefore, it is crucial to flip minimum number of bits in order not to overburden the error correction decoder. We devise a constrained combinatorial formulation for minimizing the number of flipped bits for a given set of harmful patterns. The generalized belief propagation algorithm is used to find an approximate solution for the problem. We evaluate the performance gain of our proposed approach on a data-dependent 2-D channel, where 2-D isolated-bits patterns are the harmful patterns for the channel. Furthermore, the performance of the DBF method is compared with classical 2-D constrained coding schemes for the 2-D no isolated-bits constraint on a memoryless binary symmetric channel.

2-D LDPC Codes and Joint Detection and Decoding for Two-Dimensional Magnetic Recording

Two-dimensional magnetic recording (TDMR) is a promising technology for boosting areal densities (ADs) using sophisticated signal processing algorithms within a systems framework. The read/write channel architectures have to effectively tackle 2-D inter-symbol interference (ISI), 2-D synchronization errors, media and electronic noise sources, as well as thermal asperities resulting in burst erasures. The 1-D low-density parity check (LDPC) codes are well studied to correct large 1-D burst errors/erasures. However, such 1-D LDPC codes are not suitable for correcting 2-D burst errors/erasures due to the 2-D span of errors. In this paper, we propose construction of a native 2-D LDPC code to effectively correct 2-D burst erasures. We also propose a joint detection and decoding engine based on the generalized belief propagation algorithm to simultaneously handle 2-D ISI, as well as correct bit/burst errors for TDMR channels. This paper is novel in two aspects: 1) we propose the construction of native 2-D LDPC codes to correct large 2-D burst erasures and 2) we develop a 2-D joint signal detection-decoder engine that incorporates 2-D ISI constraints, and modulation code constrains along with LDPC decoding. The native 2-D LDPC code can correct >20% more burst erasures compared with the 1-D LDPC code over a 128 $\times $ 256 2-D page of detected bits. Also, the proposed algorithm is observed to achieve a signal-to-noise ratio gain of >0.5 dB in bit error rate performance (translating to 10% increase in ADs around the 1.8 Tb/in2 regime with grain sizes of 9 nm) as compared with a decoupled detector-decoder system configuration over a small 2-D LDPC code of size 16 $\times $ 16. The efficacy of our proposed algorithm and system architecture is evaluated by assessing AD gains via simulations for a TDMR configuration comprising of a 2-D generalized partial response over the Voronoi media model assuming perfect 2-D synchronization.

Gauge-free cluster variational method by maximal messages and moment matching.

We present an implementation of the cluster variational method (CVM) as a message passing algorithm. The kind of message passing algorithm used for CVM, usually named generalized belief propagation (GBP), is a generalization of the belief propagation algorithm in the same way that CVM is a generalization of the Bethe approximation for estimating the partition function. However, the connection between fixed points of GBP and the extremal points of the CVM free energy is usually not a one-to-one correspondence because of the existence of a gauge transformation involving the GBP messages. Our contribution is twofold. First, we propose a way of defining messages (fields) in a generic CVM approximation, such that messages arrive on a given region from all its ancestors, and not only from its direct parents, as in the standard parent-to-child GBP. We call this approach maximal messages. Second, we focus on the case of binary variables, reinterpreting the messages as fields enforcing the consistency between the moments of the local (marginal) probability distributions. We provide a precise rule to enforce all consistencies, avoiding any redundancy, that would otherwise lead to a gauge transformation on the messages. This moment matching method is gauge free, i.e., it guarantees that the resulting GBP is not gauge invariant. We apply our maximal messages and moment matching GBP to obtain an analytical expression for the critical temperature of the Ising model in general dimensions at the level of plaquette CVM. The values obtained outperform Bethe estimates, and are comparable with loop corrected belief propagation equations. The method allows for a straightforward generalization to disordered systems.

High Density Turbo TDMR Detection With Local Area Influence Probabilistic Model

This paper proposes probabilistic message-passing turbo detection algorithms for 2-D magnetic recording that locally estimate magnetic grain interactions with coded data bits, and thus iteratively assist channel decoding. Such local area influence probabilistic (LAIP) techniques are especially effective at densities ranging from about 4 magnetic grains per coded bit (GPB) down to about 1 GPB, where interaction between grains and bits is significant and occasionally a bit will not be written on any grain, and hence will effectively be “overwritten” by bits on surrounding grains. By modeling the interaction among grains, LAIP enables detection of both overwritten bits and wrong-sign non-overwritten bits so that their log-likelihood ratios are assigned small magnitudes for overwritten bits and correct signs for non-overwritten bits. Simulation results with a random Voronoi magnetic media model show that the LAIP-based detector can accurately detect both overwritten bits and severely influenced non-overwritten bits, and that higher user densities and lower bit error rates can be achieved compared with our previously proposed generalized belief propagation (GBP) detector presented at the 2015 Magnetic Recording Conference (TMRC 2015). In addition, the LAIP-based detector's computer run time is 10 000 times faster than the GBP-based detector.

On one-step replica symmetry breaking in the Edwards–Anderson spin glass model

We consider a one-step replica symmetry breaking description of the Edwards–Anderson spin glass model in 2D. The ingredients of this description are a Kikuchi approximation to the free energy and a second-level statistical model built on the extremal points of the Kikuchi approximation, which are also fixed points of a generalized belief propagation (GBP) scheme. We show that a generalized free energy can be constructed where these extremal points are exponentially weighted by their Kikuchi free energy and a Parisi parameter y, and that the Kikuchi approximation of this generalized free energy leads to second-level, one-step replica symmetry breaking (1RSB), GBP equations. We then proceed analogously to the Bethe approximation case for tree-like graphs, where it has been shown that 1RSB belief propagation equations admit a survey propagation solution. We discuss when and how the one-step-replica symmetry breaking GBP equations that we obtain also allow a simpler class of solutions which can be interpreted as a class of generalized survey propagation equations for the single instance graph case.

Cycle-Based Cluster Variational Method for Direct and Inverse Inference

We elaborate on the idea that loop corrections to belief propagation could be dealt with in a systematic way on pairwise Markov random fields, by using the elements of a cycle basis to define region in a generalized belief propagation setting. The region graph is specified in such a way as to avoid dual loops as much as possible, by discarding redundant Lagrange multipliers, in order to facilitate the convergence, while avoiding instabilities associated to minimal factor graph construction. We end up with a two-level algorithm, where a belief propagation algorithm is run alternatively at the level of each cycle and at the inter-region level. The inverse problem of finding the couplings of a Markov random field from empirical covariances can be addressed region wise. It turns out that this can be done efficiently in particular in the Ising context, where fixed point equations can be derived along with a one-parameter log likelihood function to minimize. Numerical experiments confirm the effectiveness of these considerations both for the direct and inverse MRF inference.

Dynamic Grain State Estimation for High-Density TDMR: Progress and Future Directions

Dynamic grain state estimation (DGSE) algorithms for 2-D magnetic recording (TDMR) employ probabilistic message-passing algorithms that jointly estimate magnetic grain configurations and coded data bits, in order to iteratively assist channel decoding. At high densities (e.g., between 1 and 3 magnetic grains per coded bit), occasionally, a bit will not be written on any grain, and hence will effectively be overwritten (or erased) by bits on surrounding grains. DGSE enables the detection of overwritten bits so that their log-likelihood ratios are assigned small magnitudes, effectively making them erasures, which are easily filled in by linear channel codes. Past papers employing Bahl-Cocke-Jelinek-Raviv-based detectors on a simple four-rectangular-grain model have shown that the DGSE is surprisingly resilient to mismatch between the detector’s assumed grain model and the actual model generating the data. This paper presents a novel DGSE–TDMR detector based on the generalized belief propagation (GBP) algorithm. The new detector employs a random discretized-nuclei Voronoi grain model. Simulation results show that the GBP-based TDMR turbo-detector accurately detects the overwritten bits and that it achieves low decoded bit error rates at densities as high as 0.4966 user bits per grain.

Generalized belief propagation for the magnetization of the simple cubic Ising model

A new approximation of the cluster variational method is introduced for the three-dimensional Ising model on the simple cubic lattice. The maximal cluster is, as far as we know, the largest ever used in this method. A message-passing algorithm, generalized belief propagation, is used to minimize the variational free energy. Convergence properties and performance of the algorithm are investigated.The approximation is used to compute the spontaneous magnetization, which is then compared to previous results. Using the present results as the last step in a sequence of three cluster variational approximations, an extrapolation is obtained which captures the leading critical behavior with a good accuracy.

Simplifying generalized belief propagation on redundant region graphs

The cluster variation method has been developed into a general theoretical framework for treating short-range correlations in many-body systems after it was first proposed by Kikuchi in 1951. On the numerical side, a message-passing approach called generalized belief propagation (GBP) was proposed by Yedidia, Freeman and Weiss about a decade ago as a way of computing the minimal value of the cluster variational free energy and the marginal distributions of clusters of variables. However the GBP equations are often redundant, and it is quite a non-trivial task to make the GBP iteration converges to a fixed point. These drawbacks hinder the application of the GBP approach to finite-dimensional frustrated and disordered systems.In this work we report an alternative and simple derivation of the GBP equations starting from the partition function. Based on this derivation we propose a natural and systematic way of removing the redundance of the GBP equations. We apply the simplified generalized belief propagation (SGBP) equations to the two-dimensional and the three-dimensional ferromagnetic Ising model and Edwards-Anderson spin glass model. The numerical results confirm that the SGBP message-passing approach is able to achieve satisfactory performance on these model systems. We also suggest that a subset of the SGBP equations can be neglected in the numerical iteration process without affecting the final results.

Generalized Belief Propagation Detector for TDMR Microcell Model

Signal processing in TDMR encounters several challenges such as read channel modeling and detection in the presence of severe two-dimensional intersymbol interference (2-D ISI). The contribution of this paper is twofold. 1) In this paper, we introduce a novel 2-D read channel model which we call the 2-D Microcell model. In this model, we use generalized 2-D microtracks called microcells to captures the properties of irregular grain boundaries of the medium in a relatively simple yet accurate manner. The data dependent noise (DDN) distributions are analytically derived for this model. The derivation of the DDN distributions makes the 2-D Microcell suitable for detector design purposes. 2) We propose a new framework for designing truly two-dimensional detectors for the Microcell model based on near-optimal generalized belief propagation (GBP). The GBP algorithm is purposefully applied for detection in this model in order to handle the data dependent media noise which is caused by irregular bit transitions in both dimensions. Results are provided to show that the incorporation of the DDN distributions into the GBP detection helps improving the detection performance.

Nonparametric generalized belief propagation based on pseudo-junction tree for cooperative localization in wireless networks

Non-parametric belief propagation (NBP) is a well-known message passing method for cooperative localization in wireless networks. However, due to the over-counting problem in the networks with loops, NBP’s convergence is not guaranteed, and its estimates are typically less accurate. One solution for this problem is non-parametric generalized belief propagation based on junction tree. However, this method is intractable in large-scale networks due to the high-complexity of the junction tree formation, and the high-dimensionality of the particles. Therefore, in this article, we propose the non-parametric generalized belief propagation based on pseudo-junction tree (NGBP-PJT). The main difference comparing with the standard method is the formation of pseudo-junction tree, which represents the approximated junction tree based on thin graph. In addition, in order to decrease the number of high-dimensional particles, we use more informative importance density function, and reduce the dimensionality of the messages. As by-product, we also propose NBP based on thin graph (NBP-TG), a cheaper variant of NBP, which runs on the same graph as NGBP-PJT. According to our simulation and experimental results, NGBP-PJT method outperforms NBP and NBP-TG in terms of accuracy, computational, and communication cost in reasonably sized networks.

Generalized Belief Propagation for the Noiseless Capacity and Information Rates of Run-Length Limited Constraints

The performance of the generalized belief propagation algorithm to compute the noiseless capacity and mutual information rates of finite-size two-dimensional and three-dimensional run-length limited constraints is investigated. In both cases, the problem is reduced to estimating the partition function of graphical models with cycles. The partition function is then estimated using the region-based free energy approximation technique. For each constraint, a method is proposed to choose the basic regions and to construct the region graph which provides the graphical framework to run the generalized belief propagation algorithm. Simulation results for the noiseless capacity of different constraints as a function of the size of the channel are reported. In the cases that tight lower and upper bounds on the Shannon capacity exist, convergence to the Shannon capacity is discussed. For noisy constrained channels, simulation results are reported for mutual information rates as a function of signal-to-noise ratio.

Acceleration Strategies in Generalized Belief Propagation

Generalized belief propagation is a popular algorithm to perform inference on large-scale Markov random fields (MRFs) networks. This paper proposes the method of accelerated generalized belief propagation with three strategies to reduce the computational effort. First, a min-sum messaging scheme and a caching technique are used to improve the accessibility. Second, a direction set method is used to reduce the complexity of computing clique messages from quartic to cubic. Finally, a coarse-to-fine hierarchical state-space reduction method is presented to decrease redundant states. The results show that a combination of these strategies can greatly accelerate the inference process in large-scale MRFs. For common stereo matching, it results in a speed-up of about 200 times.

Characterizing and improving generalized belief propagation algorithms on the 2DEdwards–Anderson model

We study the performance of different message passing algorithms in the two-dimensionalEdwards–Anderson model. We show that the standard belief propagation (BP) algorithmconverges only at high temperature to a paramagnetic solution. Then, we test a generalizedbelief propagation (GBP) algorithm, derived from a cluster variational method(CVM) at the plaquette level. We compare its performance with BP and withother algorithms derived under the same approximation: double loop (DL) and atwo-way message passing algorithm (HAK). The plaquette-CVM approximationimproves BP in at least three ways: the quality of the paramagnetic solution at hightemperatures, a better estimate (lower) for the critical temperature, and the factthat the GBP message passing algorithm converges also to nonparamagneticsolutions. The lack of convergence of the standard GBP message passing algorithm atlow temperatures seems to be related to the implementation details and not tothe appearance of long range order. In fact, we prove that a gauge invarianceof the constrained CVM free energy can be exploited to derive a new messagepassing algorithm which converges at even lower temperatures. In all its region ofconvergence this new algorithm is faster than HAK and DL by some orders ofmagnitude.

Improved Generalized Belief Propagation for Vision Processing

Generalized belief propagation (GBP) is a region‐based belief propagation algorithm which can get good convergence in Markov random fields. However, the computation time is too heavy to use in practical engineering applications. This paper proposes a method to accelerate the efficiency of GBP. A caching technique and chessboard passing strategy are used to speed up algorithm. Then, the direction set method which is used to reduce the complexity of computing clique messages from quadric to cubic. With such a strategy the processing speed can be greatly increased. Besides, it is the first attempt to apply GBP for solving the stereomatching problem. Experiments show that the proposed algorithm can speed up by 15+ times for typical stereo matching problem and infer a more plausible result.