Standard Belief Propagation Research Articles

Counting the number of solutions in satisfiability problems with tensor-network message passing.

We investigate how to count the number of solutions in the Boolean satisfiability (SAT) problem, a fundamental problem in theoretical computer science that has applications in various domains. We convert the problem to a spin-glass problem in statistical physics and approximately compute the entropy of the problem, which corresponds to the logarithm of the number of solutions. We propose a new method for the entropy computing problem based on a combination of the tensor network method and the message-passing algorithm. The significance of the proposed method is its ability to consider the effects of both long loops and short loops present in the factor graph of the SAT problem. We validate the efficacy of our approach using 3-SAT problems defined on the random graphs and structured graphs, and show that the proposed method gives more accurate results on the number of solutions than the standard belief propagation algorithms. We also discuss the applications of our method across a wide range of combinatorial optimization problems.

Tensor Network Message Passing.

When studying interacting systems, computing their statistical properties is a fundamental problem in various fields such as physics, applied mathematics, and machine learning. However, this task can be quite challenging due to the exponential growth of the state space as the system size increases. Many standard methods have significant weaknesses. For instance, message-passing algorithms can be inaccurate and even fail to converge due to short loops, while tensor network methods can have exponential computational complexity in large graphs due to long loops. In this Letter, we propose a new method called "tensor network message passing." This approach allows us to compute local observables like marginal probabilities and correlations by combining the strengths of tensor networks in contracting small subgraphs with many short loops and the strengths of message-passing methods in globally sparse graphs, thus addressing the crucial weaknesses of both approaches. Our algorithm is exact for systems that are globally treelike and locally dense-connected when the dense local graphs have a limited tree width. We have conducted numerical experiments on synthetic and real-world graphs to compute magnetizations of Ising models and spin glasses, and have demonstrated the superiority of our approach over standard belief propagation and the recently proposed loopy message-passing algorithm. In addition, we discuss the potential applications of our method in inference problems in networks, combinatorial optimization problems, and decoding problems in quantum error correction.

An iterative BP-CNN decoder for optical fiber communication systems.

The conventional belief propagation (BP) of the low-density parity-check (LDPC) is designed based on additive white Gaussian noise (AWGN) close to the Shannon limit; however, the correlated noise due to chromatic dispersion or square-law detection results in a performance penalty in the intensity modulation and direct-detection (IM/DD) system. We propose an iterative BP cascaded convolution neural network (CNN) decoder to mitigate the correlated channel noise. We use a model of correlated Gaussian noise to verify that the noise correlation can be identified by the CNN and the decoding performance is improved by the iterative processing. We successfully demonstrate the proposed method in a 50-Gb/s 4-ary pulse amplitude modulation (PAM-4) IM/DD system. The simulation results show that the proposed decoder can achieve a BER performance improvement which is robust to transmission distance and launch optical power. The experimental results show that the iterative BP-CNN decoder outperforms the standard BP decoder by 1.2 dB in received optical power over 25-km SSMF.

Convex combination belief propagation

We present new message passing algorithms for performing inference with graphical models. Our methods are designed for the most difficult inference problems where loopy belief propagation and other heuristics fail to converge. Belief propagation is guaranteed to converge when the underlying graphical model is acyclic, but can fail to converge and is sensitive to initialization when the underlying graph has complex topology. This paper describes modifications to the standard belief propagation algorithms that lead to methods that converge to unique solutions on graphical models with arbitrary topology and potential functions.

Inference With Aggregate Data in Probabilistic Graphical Models: An Optimal Transport Approach

We consider inference (filtering) problems over probabilistic graphical models with aggregate data generated by a large population of individuals. We propose a new efficient belief propagation type algorithm over tree graphs with polynomial computational complexity as well as a global convergence guarantee. This is in contrast to previous methods that either exhibit prohibitive complexity as the population grows or do not guarantee convergence. Our method is based on optimal transport, or more specifically, multimarginal optimal transport theory. In particular, we consider an inference problem with aggregate observations, that can be seen as a structured multimarginal optimal transport problem where the cost function decomposes according to the underlying graph. Consequently, the celebrated Sinkhorn/iterative scaling algorithm for multi-marginal optimal transport can be leveraged together with the standard belief propagation algorithm to establish an efficient inference scheme which we call Sinkhorn belief propagation (SBP). We further specialize the SBP algorithm to cases associated with hidden Markov models due to their significance in control and estimation. We demonstrate the performance of our algorithm on applications such as inferring population flow from aggregate observations. We also show that in the special case where the aggregate observations are in Dirac form, our algorithm naturally reduces to the standard belief propagation algorithm.

Degenerate Quantum LDPC Codes With Good Finite Length Performance

We study the performance of medium-length quantum LDPC (QLDPC) codes in the depolarizing channel. Only degenerate codes with the maximal stabilizer weight much smaller than their minimum distance are considered. It is shown that with the help of OSD-like post-processing the performance of the standard belief propagation (BP) decoder on many QLDPC codes can be improved by several orders of magnitude. Using this new BP-OSD decoder we study the performance of several known classes of degenerate QLDPC codes including hypergraph product codes, hyperbicycle codes, homological product codes, and Haah's cubic codes. We also construct several interesting examples of short generalized bicycle codes. Some of them have an additional property that their syndromes are protected by small BCH codes, which may be useful for the fault-tolerant syndrome measurement. We also propose a new large family of QLDPC codes that contains the class of hypergraph product codes, where one of the used parity-check matrices is square. It is shown that in some cases such codes have better performance than hypergraph product codes. Finally, we demonstrate that the performance of the proposed BP-OSD decoder for some of the constructed codes is better than for a relatively large surface code decoded by a near-optimal decoder.

Learning to Decode Polar Codes With One-Bit Quantizer

A deep learning method for improving the performance of polar belief propagation (BP) decoder equipped with a one-bit quantizer is proposed. The method generalizes the standard polar BP algorithm by assigning weights to the layers of the unfolded factor graph. These weights can be learned autonomously using deep learning techniques. We prove that the improved polar BP decoder has a symmetric structure, so that the weights can be trained by an all-zero codeword rather than an exponential number of codewords. In order to accelerate the training convergence, a layer-based weight assignment scheme is designed, which decreases the amount of trainable weights. Simulation results show that the improved polar BP decoder with a one-bit quantizer outperforms the standard polar BP decoder with a 2-bit quantizer and achieves faster convergence.

Incremental inference of collective graphical models

We consider incremental inference problems from aggregate data for collective dynamics. In particular, we address the problem of estimating the aggregate marginals of a Markov chain from noisy aggregate observations in an incremental (online) fashion. We propose a sliding window Sinkhorn belief propagation (SW-SBP) algorithm that utilizes a sliding window filter of the most recent noisy aggregate observations along with encoded information from discarded observations. Our algorithm is built upon the recently proposed multi-marginal optimal transport based SBP algorithm that leverages standard belief propagation and Sinkhorn algorithm to solve inference problems from aggregate data. We demonstrate the performance of our algorithm on applications such as inferring population flow from aggregate observations.

From Support Propagation to Belief Propagation in Constraint Programming


 
 
 The distinctive driving force of constraint programming to solve combinatorial problems has been a privileged access to problem structure through the high-level models it uses. From that exposed structure in the form of so-called global constraints, powerful inference algorithms have shared information between constraints by propagating it through shared variables’ domains, traditionally by removing unsupported values. This paper investigates a richer propagation medium made possible by recent work on counting solutions inside constraints. Beliefs about individual variable-value assignments are exchanged between contraints and iteratively adjusted. It generalizes standard support propagation and aims to converge to the true marginal distributions of the solutions over individual variables. Its advantage over standard belief propagation is that the higher-level models featuring large-arity (global) constraints do not tend to create as many cycles, which are known to be problematic for convergence. The necessary architectural changes to a constraint programming solver are described and an empirical study of the proposal is conducted on its implementation. We find that it provides close approximations to the true marginals and that it significantly improves search guidance.
 
 

One-view occlusion detection for stereo matching with a fully connected CRF model.

In this paper, we extend the standard belief propagation (BP) sequential technique proposed in the tree-reweighted sequential method [15] to the fully connected CRF models with the geodesic distance affinity. The proposed method has been applied to the stereo matching problem. Also a new approach to the BP marginal solution is proposed that we call one-view occlusion detection (OVOD). In contrast to the standard winner takes all (WTA) estimation, the proposed OVOD solution allows to find occluded regions in the disparity map and simultaneously improve the matching result. As a result we can perform only one energy minimization process and avoid the cost calculation for the second view and the left-right check procedure. We show that the OVOD approach considerably improves results for cost augmentation and energy minimization techniques in comparison with the standard one-view affinity space implementation. We apply our method to the Middlebury data set and reach state-of-the-art especially for median, average and mean squared error metrics.

Novel Belief Propagation Algorithm for Stereo Matching With a Robust Cost Computation

Stereo matching is a challenging problem in the field of computer vision and has recently received extensive attention. However, the traditional methods are labor intensive and premised on specific conditions. In this paper, we propose a robust stereo matching cost algorithm that relies on refined features extracted by a stack auto-encoder. These features are robust for different types of image pairs, which significantly improve the generality of the proposed matching algorithm. In addition, we smoothed the belief volume with a guided filter to improve the performance of the belief propagation algorithm on edge regions. To deal with the time consumption issue of the standard belief propagation algorithm, we also proposed a constraint method to speed it up. This constraint does not degrade the matching results and can easily be generalized to other message passing algorithms. The experiments conducted on a Middlebury benchmark dataset demonstrate the effectiveness of the proposed algorithms.

Scaling factor optimization of turbo-polar iterative decoding

In this paper, we investigate the weighted iterative decoding to improve the performance of turbo-polar code. First of all, a minimum weighted mean square error criterion is proposed to optimize the scaling factors (SFs). Secondly, for two typical iterative algorithms, such as soft cancellation (SCAN) and belief propagation (BP) decoding, genie-aided decoders are proposed as the ideal reference of the practical decoding. Guided by this optimization framework, the optimal SFs of SCAN or BP decoders are obtained. The bit error rate performance of turbo-polar code with the optimal SFs can achieve 0.3 dB or 0.7 dB performance gains over the standard SCAN or BP decoding respectively.

Deep Learning Methods for Improved Decoding of Linear Codes

The problem of low complexity, close to optimal, channel decoding of linear codes with short to moderate block length is considered. It is shown that deep learning methods can be used to improve a standard belief propagation decoder, despite the large example space. Similar improvements are obtained for the min-sum algorithm. It is also shown that tying the parameters of the decoders across iterations, so as to form a recurrent neural network architecture, can be implemented with comparable results. The advantage is that significantly less parameters are required. We also introduce a recurrent neural decoder architecture based on the method of successive relaxation. Improvements over standard belief propagation are also observed on sparser Tanner graph representations of the codes. Furthermore, we demonstrate that the neural belief propagation decoder can be used to improve the performance, or alternatively reduce the computational complexity, of a close to optimal decoder of short BCH codes.

An Iterative BP-CNN Architecture for Channel Decoding

Inspired by recent advances in deep learning, we propose a novel iterative BP-CNN architecture for channel decoding under correlated noise. This architecture concatenates a trained convolutional neural network (CNN) with a standard belief-propagation (BP) decoder. The standard BP decoder is used to estimate the coded bits, followed by a CNN to remove the estimation errors of the BP decoder and obtain a more accurate estimation of the channel noise. Iterating between BP and CNN will gradually improve the decoding SNR and hence result in better decoding performance. To train a well-behaved CNN model, we define a new loss function which involves not only the accuracy of the noise estimation but also the normality test for the estimation errors, i.e., to measure how likely the estimation errors follow a Gaussian distribution. The introduction of the normality test to the CNN training shapes the residual noise distribution and further reduces the BER of the iterative decoding, compared to using the standard quadratic loss function. We carry out extensive experiments to analyze and verify the proposed framework. The iterative BP-CNN decoder has better BER performance with lower complexity, is suitable for parallel implementation, does not rely on any specific channel model or encoding method, and is robust against training mismatches. All of these features make it a good candidate for decoding modern channel codes.

Improved Belief Propagation Decoding Algorithm for Short Polar Codes

In this paper, we discuss the belief propagation (BP) decoding of polar codes. The performance of polar codes for short lengths is not satisfactory. Therefore, motivated by this we propose a novel technique to improve the performance of polar codes under BP decoding. In order to enhance the reliability of the variable nodes’ propagated messages in BP decoding, the messages are multiplied by previous messages of variable nodes. It is also shown that the BP decoding can be performed on polar codes if the factor graph is constructed according to the parity check matrix of polar codes. Simulation results in binary-input additive white Gaussian noise channel show that using the proposed method a gain of 1–2 dB can be achieved. We also show that the complexity of proposed decoder is same as the complexity of standard BP decoder. Furthermore, we show that the proposed decoder requires about 10–25 iterations less than the standard BP decoder.

Context guided belief propagation for remote sensing image classification.

We propose a context guided belief propagation (BP) algorithm to perform high spatial resolution multispectral imagery (HSRMI) classification efficiently utilizing superpixel representation. One important characteristic of HSRMI is that different land cover objects possess a similar spectral property. This property is exploited to speed up the standard BP (SBP) in the classification process. Specifically, we leverage this property of HSRMI as context information to guide messages passing in SBP. Furthermore, the spectral and structural features extracted at the superpixel level are fed into a Markov random field framework to address the challenge of low interclass variation in HSRMI classification by minimizing the discrete energy through context guided BP (CBP). Experiments show that the proposed CBP is significantly faster than the SBP while retaining similar performance as compared with SBP. Compared to the baseline methods, higher classification accuracy is achieved by the proposed CBP when the context information is used with both spectral and structural features.

이진 순환 부호를 쓰는 GLDPC 부호의 수평-수직 결합 직렬 복호

It is well known that serial belief propagation (BP) decoding for low-density parity-check (LDPC) codes achieves faster convergence without any increase of decoding complexity per iteration and bit error rate (BER) performance loss than standard parallel BP (PBP) decoding. Serial BP (SBP) decoding, such as horizontal SBP (H-SBP) decoding or vertical SBP (V-SBP) decoding, updates check nodes or variable nodes faster than standard PBP decoding within a single iteration. In this paper, we propose combined horizontal-vertical SBP (CHV-SBP) decoding. By the same reasoning, CHV-SBP decoding updates check nodes or variable nodes faster than SBP decoding within a serialized step in an iteration. CHV-SBP decoding achieves faster convergence than H-SBP or V-SBP decoding. We compare these decoding schemes in details. We also show in simulations that the convergence rate, in iterations, for CHV-SBP decoding is about <TEX>$\frac{1}{6}$</TEX> of that for standard PBP decoding, while the convergence rate for SBP decoding is about <TEX>$\frac{1}{2}$</TEX> of that for standard PBP decoding. In simulations, we use recently proposed generalized LDPC (GLDPC) codes with binary cyclic codes (BCC).

Tree-Structured Expectation Propagation for LDPC Decoding over BMS Channels

In this paper, we put forward the tree-structured expectation propagation (TEP) algorithm for decoding block and convolutional low-density parity-check codes over any binary channel. We have already shown that TEP improves belief propagation (BP) over the binary erasure channel (BEC) by imposing marginal constraints over a set of pairs of variables that form a tree or a forest. The TEP decoder is a message-passing algorithm that sequentially builds a tree/forest of erased variables to capture additional information disregarded by the standard BP decoder, which leads to a noticeable reduction of the error rate for finite-length codes. In this paper, we show how the TEP can be extended to any channel, specifically to binary memoryless symmetric (BMS) channels. We particularly focus on how the TEP algorithm can be adapted for any channel model and, more importantly, how to choose the tree/forest to keep the gains observed for block and convolutional LDPC codes over the BEC.

Graph‐based low complexity detection algorithms in multiple‐input–multiple‐out systems: an edge selection approach

In this study, the problem of low complexity multiple-input-multiple-out signal detection based on belief propagation (BP) is addressed. The authors propose an edge selection approach that works on factor graph model to cut down the number of circles and high complexity of standard BP algorithm. The message passing from factor nodes to variable nodes is updated by only partial edges, and the mean feedback method is designed to compensate the information loss brought by the edge selection. Both binary and high-order modulations are considered, and the scheme of mapping between bit soft output and modulation symbols when computing the feedback information is discussed. In addition, a minimum mean-square error filter initialised algorithm is proposed, in which the initial message of BP detection is exploited. Both binary and high-order modulations are discussed as well when the authors design this initial message. Simulation results along with convergence and complexity analyses verify that the proposed edge selection approach can achieve good performance with low complexity, and significantly outperform the existing methods with comparative complexity. Moreover, our approach has asymptotic optimality and is a self-adapting scheme, which can achieve the trade-off between performance and complexity by varying the number of selected edges.

Performance Analysis of Iterative Decoding Algorithms with Memory over Memoryless Channels

Density evolution is often used to determine the performance of an ensemble of low-density parity-check (LDPC) codes under iterative message-passing algorithms. Conventional density evolution techniques over memoryless channels are based on the assumption that messages at iteration l are only a function of the messages at iteration l -1 and possibly the channel output. This assumption is valid for many algorithms such as standard belief propagation (BP) and min-sum (MS) algorithms. However, there are other important iterative algorithms such as successive relaxation (SR) versions of BP and MS, and differential decoding with binary message passing (DD-BMP) algorithm of Mobini et al., for which this assumption is not valid. The reason is the introduction of memory in these algorithms. In this work, we propose a model for iterative decoding algorithms with memory which covers SR and DD-BMP algorithms as special cases. Based on this model, we derive a Bayesian network for iterative algorithms with memory over memoryless channels and use this representation to analyze the performance of the algorithms using density evolution. The density evolution technique is developed based on truncating the memory of the decoding process and approximating it with a finite order Markov process, and can be implemented efficiently. As an example, we apply our technique to analyze the performance of DD-BMP on regular LDPC code ensembles, and make a number of interesting observations with regard to the performance/complexity tradeoff of DD-BMP in comparison with BP and MS algorithms. The model presented in this paper is based on certain simplifying assumptions about the memory structure of iterative algorithms such as the existence of memory only at the output of variable nodes in the code's Tanner graph rather than at both outputs of variable and check nodes. The Bayesian network framework introduced here however, can still be used to analyze the more general scenarios.