Generalized Belief Propagation Algorithm Research Articles

Generalized Belief Propagation Algorithms for Decoding of Surface Codes

Belief propagation (BP) is well-known as a low complexity decoding algorithm with a strong performance for important classes of quantum error correcting codes, e.g. notably for the quantum low-density parity check (LDPC) code class of random expander codes. However, it is also well-known that the performance of BP breaks down when facing topological codes such as the surface code, where naive BP fails entirely to reach a below-threshold regime, i.e. the regime where error correction becomes useful. Previous works have shown, that this can be remedied by resorting to post-processing decoders outside the framework of BP. In this work, we present a generalized belief propagation method with an outer re-initialization loop that successfully decodes surface codes, i.e. opposed to naive BP it recovers the sub-threshold regime known from decoders tailored to the surface code and from statistical-mechanical mappings. We report a threshold of 17% under independent bit-and phase-flip data noise (to be compared to the ideal threshold of 20.6%) and a threshold value of 14% under depolarizing data noise (compared to the ideal threshold of 18.9%), which are on par with thresholds achieved by non-BP post-processing methods.

Performing Bayesian Network Inference Using Amortized Region Approximation with Graph Factorization

Exact inference for large, directed graphical models, also known as Bayesian networks (BNs), can be intractable as the space complexity grows exponentially in the tree-width of the model. Approximate inference, such as generalized belief propagation (GBP), is used instead. GBP treats inference as the Bethe/Kikuchi energy function optimization problem. The solution is found using iterative message passing, which is inefficient and convergent problematic. Recent progress on amortized technique for GBP is an attractive alternative solution that can optimize the Bethe/Kikuchi energy function using (deep) neural networks, requiring no message passing. Despite being efficient, the amortized technique for GBP is applied to undirected graphical models with specific structures and factors, with no guarantee of the approximation quality. This is because the energy function to be amortized is defined by a region (or factor) graph that is ad hoc and difficult to construct to ensure sensible approximations. This paper proposes a new amortized GBP algorithm applied to BN for efficient inference. The proposed algorithm is composed of the following: (i) a new pairwise conversion (PWC) algorithm that converts all the conditional probability distributions in the BN into pairwise factors to facilitate efficient region graph constructions; (ii) following PWC, an improved loop structured region graph (LSRG) algorithm was derived to generate a valid region graph satisfying desired regional properties; and (iii) the energy function defined by the proposed PWC-LSRG region graph can be directly amortized using (deep) neural networks to ensure sensible approximations. Empirical studies show that the proposed amortized PWC-LSRG algorithm is of practical use and significantly improves convergence and efficiency compared to conventional algorithms.

A Deliberate Bit Flipping Coding Scheme for Data-Dependent Two-Dimensional Channels

In this paper, we present a deliberate bit flipping (DBF) coding scheme for binary two-dimensional (2-D) channels, where specific patterns in channel inputs are the significant cause of errors. The idea is to eliminate a constrained encoder and, instead, embed a constraint into an error correction codeword that is arranged into a 2-D array by deliberately flipping the bits that violate the constraint. The DBF method relies on the error correction capability of the code being used so that it should be able to correct both deliberate errors and channel errors. Therefore, it is crucial to flip minimum number of bits in order not to overburden the error correction decoder. We devise a constrained combinatorial formulation for minimizing the number of flipped bits for a given set of harmful patterns. The generalized belief propagation algorithm is used to find an approximate solution for the problem. We evaluate the performance gain of our proposed approach on a data-dependent 2-D channel, where 2-D isolated-bits patterns are the harmful patterns for the channel. Furthermore, the performance of the DBF method is compared with classical 2-D constrained coding schemes for the 2-D no isolated-bits constraint on a memoryless binary symmetric channel.

2-D LDPC Codes and Joint Detection and Decoding for Two-Dimensional Magnetic Recording

Two-dimensional magnetic recording (TDMR) is a promising technology for boosting areal densities (ADs) using sophisticated signal processing algorithms within a systems framework. The read/write channel architectures have to effectively tackle 2-D inter-symbol interference (ISI), 2-D synchronization errors, media and electronic noise sources, as well as thermal asperities resulting in burst erasures. The 1-D low-density parity check (LDPC) codes are well studied to correct large 1-D burst errors/erasures. However, such 1-D LDPC codes are not suitable for correcting 2-D burst errors/erasures due to the 2-D span of errors. In this paper, we propose construction of a native 2-D LDPC code to effectively correct 2-D burst erasures. We also propose a joint detection and decoding engine based on the generalized belief propagation algorithm to simultaneously handle 2-D ISI, as well as correct bit/burst errors for TDMR channels. This paper is novel in two aspects: 1) we propose the construction of native 2-D LDPC codes to correct large 2-D burst erasures and 2) we develop a 2-D joint signal detection-decoder engine that incorporates 2-D ISI constraints, and modulation code constrains along with LDPC decoding. The native 2-D LDPC code can correct >20% more burst erasures compared with the 1-D LDPC code over a 128 $\times $ 256 2-D page of detected bits. Also, the proposed algorithm is observed to achieve a signal-to-noise ratio gain of >0.5 dB in bit error rate performance (translating to 10% increase in ADs around the 1.8 Tb/in2 regime with grain sizes of 9 nm) as compared with a decoupled detector-decoder system configuration over a small 2-D LDPC code of size 16 $\times $ 16. The efficacy of our proposed algorithm and system architecture is evaluated by assessing AD gains via simulations for a TDMR configuration comprising of a 2-D generalized partial response over the Voronoi media model assuming perfect 2-D synchronization.

Generalized Belief Propagation Detector for TDMR Microcell Model

Signal processing in TDMR encounters several challenges such as read channel modeling and detection in the presence of severe two-dimensional intersymbol interference (2-D ISI). The contribution of this paper is twofold. 1) In this paper, we introduce a novel 2-D read channel model which we call the 2-D Microcell model. In this model, we use generalized 2-D microtracks called microcells to captures the properties of irregular grain boundaries of the medium in a relatively simple yet accurate manner. The data dependent noise (DDN) distributions are analytically derived for this model. The derivation of the DDN distributions makes the 2-D Microcell suitable for detector design purposes. 2) We propose a new framework for designing truly two-dimensional detectors for the Microcell model based on near-optimal generalized belief propagation (GBP). The GBP algorithm is purposefully applied for detection in this model in order to handle the data dependent media noise which is caused by irregular bit transitions in both dimensions. Results are provided to show that the incorporation of the DDN distributions into the GBP detection helps improving the detection performance.

Generalized Belief Propagation for the Noiseless Capacity and Information Rates of Run-Length Limited Constraints

The performance of the generalized belief propagation algorithm to compute the noiseless capacity and mutual information rates of finite-size two-dimensional and three-dimensional run-length limited constraints is investigated. In both cases, the problem is reduced to estimating the partition function of graphical models with cycles. The partition function is then estimated using the region-based free energy approximation technique. For each constraint, a method is proposed to choose the basic regions and to construct the region graph which provides the graphical framework to run the generalized belief propagation algorithm. Simulation results for the noiseless capacity of different constraints as a function of the size of the channel are reported. In the cases that tight lower and upper bounds on the Shannon capacity exist, convergence to the Shannon capacity is discussed. For noisy constrained channels, simulation results are reported for mutual information rates as a function of signal-to-noise ratio.

Characterizing and improving generalized belief propagation algorithms on the 2DEdwards–Anderson model

We study the performance of different message passing algorithms in the two-dimensionalEdwards–Anderson model. We show that the standard belief propagation (BP) algorithmconverges only at high temperature to a paramagnetic solution. Then, we test a generalizedbelief propagation (GBP) algorithm, derived from a cluster variational method(CVM) at the plaquette level. We compare its performance with BP and withother algorithms derived under the same approximation: double loop (DL) and atwo-way message passing algorithm (HAK). The plaquette-CVM approximationimproves BP in at least three ways: the quality of the paramagnetic solution at hightemperatures, a better estimate (lower) for the critical temperature, and the factthat the GBP message passing algorithm converges also to nonparamagneticsolutions. The lack of convergence of the standard GBP message passing algorithm atlow temperatures seems to be related to the implementation details and not tothe appearance of long range order. In fact, we prove that a gauge invarianceof the constrained CVM free energy can be exploited to derive a new messagepassing algorithm which converges at even lower temperatures. In all its region ofconvergence this new algorithm is faster than HAK and DL by some orders ofmagnitude.

Join-Graph Propagation Algorithms.

The paper investigates parameterized approximate message-passing schemes that are based on bounded inference and are inspired by Pearl's belief propagation algorithm (BP). We start with the bounded inference mini-clustering algorithm and then move to the iterative scheme called Iterative Join-Graph Propagation (IJGP), that combines both iteration and bounded inference. Algorithm IJGP belongs to the class of Generalized Belief Propagation algorithms, a framework that allowed connections with approximate algorithms from statistical physics and is shown empirically to surpass the performance of mini-clustering and belief propagation, as well as a number of other state-of-the-art algorithms on several classes of networks. We also provide insight into the accuracy of iterative BP and IJGP by relating these algorithms to well known classes of constraint propagation schemes.

Kikuchi approximation method for joint decoding of LDPC codes and partial-response channels

In this letter, we apply the Kikuchi approximation method to the problem of joint decoding of a low-density parity-check code and a partial-response channel. The Kikuchi method is, in general, more powerful than the conventional loopy belief propagation (BP) algorithm, and can produce better approximations to an underlying inference problem. We will first review the Kikuchi approximation method and the generalized BP algorithm, which is an iterative message-passing algorithm based on this method. We will then report simulation results which show that the Kikuchi method outperforms the best conventional iterative method

Recovering intrinsic images from a single image

Interpreting real-world images requires the ability distinguish the different characteristics of the scene that lead to its final appearance. Two of the most important of these characteristics are the shading and reflectance of each point in the scene. We present an algorithm that uses multiple cues to recover shading and reflectance intrinsic images from a single image. Using both color information and a classifier trained to recognize gray-scale patterns, given the lighting direction, each image derivative is classified as being caused by shading or a change in the surface's reflectance. The classifiers gather local evidence about the surface's form and color, which is then propagated using the Generalized Belief Propagation algorithm. The propagation step disambiguates areas of the image where the correct classification is not clear from local evidence. We use real-world images to demonstrate results and show how each component of the system affects the results.

Constructing Free-Energy Approximations and Generalized Belief Propagation Algorithms

Important inference problems in statistical physics, computer vision, error-correcting coding theory, and artificial intelligence can all be reformulated as the computation of marginal probabilities on factor graphs. The belief propagation (BP) algorithm is an efficient way to solve these problems that is exact when the factor graph is a tree, but only approximate when the factor graph has cycles. We show that BP fixed points correspond to the stationary points of the approximation of the free energy for a factor graph. We explain how to obtain region-based free energy approximations that improve the approximation, and corresponding generalized belief propagation (GBP) algorithms. We emphasize the conditions a free energy approximation must satisfy in order to be a or maxent-normal approximation. We describe the relationship between four different methods that can be used to generate valid approximations: the Bethe method, the junction graph method, the cluster variation method, and the region graph method. Finally, we explain how to tell whether a region-based approximation, and its corresponding GBP algorithm, is likely to be accurate, and describe empirical results showing that GBP can significantly outperform BP.

CCCP Algorithms to Minimize the Bethe and Kikuchi Free Energies: Convergent Alternatives to Belief Propagation

This article introduces a class of discrete iterative algorithms that are provably convergent alternatives to belief propagation (BP) and generalized belief propagation (GBP). Our work builds on recent results by Yedidia, Freeman, and Weiss (2000), who showed that the fixed points of BP and GBP algorithms correspond to extrema of the Bethe and Kikuchi free energies, respectively. We obtain two algorithms by applying CCCP to the Bethe and Kikuchi free energies, respectively (CCCP is a procedure, introduced here, for obtaining discrete iterative algorithms by decomposing a cost function into a concave and a convex part). We implement our CCCP algorithms on two- and three-dimensional spin glasses and compare their results to BP and GBP. Our simulations show that the CCCP algorithms are stable and converge very quickly (the speed of CCCP is similar to that of BP and GBP). Unlike CCCP, BP will often not converge for these problems (GBP usually, but not always, converges). The results found by CCCP applied to the Bethe or Kikuchi free energies are equivalent, or slightly better than, those found by BP or GBP, respectively (when BP and GBP converge). Note that for these, and other problems, BP and GBP give very accurate results (see Yedidia et al., 2000), and failure to converge is their major error mode. Finally, we point out that our algorithms have a large range of inference and learning applications.