Application Of Belief Propagation Research Articles

Self-organization scheme for balanced routing in large-scale multi-hop networks

We propose a self-organization scheme for cost-effective and load-balanced routing in multi-hop networks. To avoid overloading nodes that provide favourable routing conditions, we assign each node with a cost function that penalizes high loads. Thus, finding routes to sink nodes is formulated as an optimization problem in which the global objective function strikes a balance between route costs and node loads. We apply belief propagation (its min-sum version) to solve the network optimization problem and obtain a distributed algorithm whereby the nodes collectively discover globally optimal routes by performing low-complexity computations and exchanging messages with their neighbours. We prove that the proposed method converges to the global optimum after a finite number of local exchanges of messages. Finally, we demonstrate numerically our framework’s efficacy in balancing the node loads and study the trade-off between load reduction and total cost minimization.

Leveraging cluster backbones for improving MAP inference in statistical relational models

A wide range of important problems in machine learning, expert system, social network analysis, bioinformatics and information theory can be formulated as a maximum a-posteriori (MAP) inference problem on statistical relational models. While off-the-shelf inference algorithms that are based on local search and message-passing may provide adequate solutions in some situations, they frequently give poor results when faced with models that possess high-density networks. Unfortunately, these situations always occur in models of real-world applications. As such, accurate and scalable maximum a-posteriori (MAP) inference on such models often remains a key challenge. In this paper, we first introduce a novel family of extended factor graphs that are parameterized by a smoothing parameter χ ∈ [0,1]. Applying belief propagation (BP) message-passing to this family formulates a new family of W eighted S urvey P ropagation algorithms (WSP-χ) applicable to relational domains. Unlike off-the-shelf inference algorithms, WSP-χ detects the “backbone” ground atoms in a solution cluster that involve potentially optimal MAP solutions: the cluster backbone atoms are not only portions of the optimal solutions, but they also can be exploited for scaling MAP inference by iteratively fixing them to reduce the complex parts until the network is simplified into one that can be solved accurately using any conventional MAP inference method. We also propose a lazy variant of this WSP-χ family of algorithms. Our experiments on several real-world problems show the efficiency of WSP-χ and its lazy variants over existing prominent MAP inference solvers such as MaxWalkSAT, RockIt, IPP, SP-Y and WCSP.

Variational BEJG Solvers for Marginal-MAP Inference with Accurate Approximation of B-Conditional Entropy

Previously proposed variational techniques for approximate MMAP inference in complex graphical models of high-order factors relax a dual variational objective function to obtain its tractable approximation, and further perform MMAP inference in the resulting simplified graphical model, where the sub-graph with decision variables is assumed to be a disconnected forest. In contrast, we developed novel variational MMAP inference algorithms and proximal convergent solvers, where we can improve the approximation accuracy while better preserving the original MMAP query by designing such a dual variational objective function that an upper bound approximation is applied only to the entropy of decision variables. We evaluate the proposed algorithms on both simulated synthetic datasets and diagnostic Bayesian networks taken from the UAI inference challenge, and our solvers outperform other variational algorithms in a majority of reported cases. Additionally, we demonstrate the important real-life application of the proposed variational approaches to solve complex tasks of policy optimization by MMAP inference, and performance of the implemented approximation algorithms is compared. Here, we demonstrate that the original task of optimizing POMDP controllers can be approached by its reformulation as the equivalent problem of marginal-MAP inference in a novel single-DBN generative model, which guarantees that the control policies computed by probabilistic inference over this model are optimal in the traditional sense. Our motivation for approaching the planning problem through probabilistic inference in graphical models is explained by the fact that by transforming a Markovian planning problem into the task of probabilistic inference (a marginal MAP problem) and applying belief propagation techniques in generative models, we can achieve a computational complexity reduction from PSPACE-complete or NEXP-complete to NPPP-complete in comparison to solving the POMDP and Dec-POMDP models respectively search vs. dynamic programming).

Weighted community detection and data clustering using message passing

Grouping objects into clusters based on the similarities or weights between them is one of the most important problems in science and engineering. In this work, by extending message-passing algorithms and spectral algorithms proposed for an unweighted community detection problem, we develop a non-parametric method based on statistical physics, by mapping the problem to the Potts model at the critical temperature of spin-glass transition and applying belief propagation to solve the marginals corresponding to the Boltzmann distribution. Our algorithm is robust to over-fitting and gives a principled way to determine whether there are significant clusters in the data and how many clusters there are. We apply our method to different clustering tasks. In the community detection problem in weighted and directed networks, we show that our algorithm significantly outperforms existing algorithms. In the clustering problem, where the data were generated by mixture models in the sparse regime, we show that our method works all the way down to the theoretical limit of detectability and gives accuracy very close to that of the optimal Bayesian inference. In the semi-supervised clustering problem, our method only needs several labels to work perfectly in classic datasets. Finally, we further develop Thouless–Anderson–Palmer equations which heavily reduce the computation complexity in dense networks but give almost the same performance as belief propagation.

Belief Propagation-Based Cognitive Routing in Maritime Ad Hoc Networks

Cognitive routing for maritime wireless ad hoc networks is proposed in this paper to find a stable path between source and destination. This is ship-to-ship communication where two ships can communicate when they not only have consensus on a common idle channel but also are in the transmission range of each other. We apply belief propagation algorithm for collaborative spectrum sensing. Every user (ship) exchanges its local decisions with its neighbors to compute the final belief about the state of the channel. These beliefs are applied for estimating the link duration for total number of hops between source and destination. Then, a path is selected which maximizes the path duration among all the paths in the network to reach the destination. We apply both flood-based and geographical routing protocols to find a route between source and destination for different scenarios. We simulate our scheme for different ocean settings and evaluate path durations for different number of ships. The results report a significant increase in path duration as the number of nodes increases in the network. In addition, we verify that path duration increases with an increase in probability of primary user being idle via extensive simulations. Hence, our scheme provides stable path selection for communication among users onboard.

Reweighted Belief Propagation and Quiet Planting for Random K-SAT

We study the random K-satisfiability problem using a partition functionwhere each solution is reweighted according to the number of variables that satisfy every clause. We apply belief propagation and the related cavity method to the reweighted partition function. This allows us to obtain several new results on the properties of random K-satisfiability problem. In particular the reweighting allows to introduce a planted ensemble that generates instances that are, in some region of parameters, equivalent to random instances. We are hence able to generate at the same time a typical random SAT instance and one of its solutions. We study the relation between clustering and belief propagation fixed points and we give a direct evidence for the existence of purely entropic (rather than energetic) barriers between clusters in some region of parameters in the random K-satisfiability problem. We exhibit, in some large planted instances, solutions with a non-trivial whitening core; such solutions were known to exist but were so far never found on very large instances. Finally, we discuss algorithmic hardness of such planted instances and we determine a region of parameters in which planting leads to satisfiable benchmarks that, up to our knowledge, are the hardest known.

Spatio-Temporally Consistent View Synthesis From Video-Plus-Depth Data With Global Optimization

We propose a novel algorithm to generate a virtual-view video from a video-plus-depth sequence. The proposed method enforces the spatial and temporal consistency in the disocclusion regions by formulating the problem as an energy minimization problem in a Markov random field (MRF) framework. At the system level, we first recover the depth images and the motion vector maps after the image warping with the preprocessed depth map. Then, we formulate the energy function for the MRF with additional shift variables for each node. To reduce the high computational complexity of applying belief propagation (BP) to this problem, we present a multilevel BPs by using BP with smaller numbers of label candidates for each level. Finally, the Poisson image reconstruction is applied to improve the color consistency along the boundary of the disocclusion region in the synthesized image. Experimental results demonstrate the performance of the proposed method on several publicly available datasets.

Human visual system and segment-based disparity estimation

In this paper, an efficient segment-based disparity estimation algorithm is proposed, based on wavelet transform and human visual system. The core idea is to estimate disparity not only from the left and right images but also from their decomposed sub-bands up to a certain level. The stereo image pair is divided into the segments of homogeneous color. Instead of assigning a disparity value to each pixel inside a segment, a disparity plane is assigned to each segment and the stereo matching problem is formulated as an energy minimization problem in the segmented domain. The optimal disparity plane labeling is approximated by applying belief propagation, which assigns the corresponding disparity plane to each segment. The obtained disparity maps are merged into a single disparity map using the human visual system model. Experiments with stereo image pairs show the validity of the proposed method.

Orientation-dependent backbone-only residue pair scoring functions for fixed backbone protein design

BackgroundEmpirical scoring functions have proven useful in protein structure modeling. Most such scoring functions depend on protein side chain conformations. However, backbone-only scoring functions do not require computationally intensive structure optimization and so are well suited to protein design, which requires fast score evaluation. Furthermore, scoring functions that account for the distinctive relative position and orientation preferences of residue pairs are expected to be more accurate than those that depend only on the separation distance.ResultsResidue pair scoring functions for fixed backbone protein design were derived using only backbone geometry. Unlike previous studies that used spherical harmonics to fit 2D angular distributions, Gaussian Mixture Models were used to fit the full 3D (position only) and 6D (position and orientation) distributions of residue pairs. The performance of the 1D (residue separation only), 3D, and 6D scoring functions were compared by their ability to identify correct threading solutions for a non-redundant benchmark set of protein backbone structures. The threading accuracy was found to steadily increase with increasing dimension, with the 6D scoring function achieving the highest accuracy. Furthermore, the 3D and 6D scoring functions were shown to outperform side chain-dependent empirical potentials from three other studies. Next, two computational methods that take advantage of the speed and pairwise form of these new backbone-only scoring functions were investigated. The first is a procedure that exploits available sequence data by averaging scores over threading solutions for homologs. This was evaluated by applying it to the challenging problem of identifying interacting transmembrane alpha-helices and found to further improve prediction accuracy. The second is a protein design method for determining the optimal sequence for a backbone structure by applying Belief Propagation optimization using the 6D scoring functions. The sensitivity of this method to backbone structure perturbations was compared with that of fixed-backbone all-atom modeling by determining the similarities between optimal sequences for two different backbone structures within the same protein family. The results showed that the design method using 6D scoring functions was more robust to small variations in backbone structure than the all-atom design method.ConclusionsBackbone-only residue pair scoring functions that account for all six relative degrees of freedom are the most accurate and including the scores of homologs further improves the accuracy in threading applications. The 6D scoring function outperformed several side chain-dependent potentials while avoiding time-consuming and error prone side chain structure prediction. These scoring functions are particularly useful as an initial filter in protein design problems before applying all-atom modeling.

An improved belief propagation method for dynamic collage

This paper presents a new photo browsing technique, dynamic collage. Although previous photo collage techniques have innate advantages for viewing several photos collectively, they only focus on a static two-dimensional arrangement of photos so that the scalability is limited. In dynamic collage, new photos are incrementally inserted into the collage one by one while the old photos are removed accordingly, the positions of photos in the canvas are updated with a local and incremental manner to form a new layout so as to maximize visibility of all the important information embedded in the current collage meanwhile maintaining the visual continuity of two successive collages. To achieve this goal, a carefully designed optimization method based on belief propagation is employed. Unlike most traditional applications of belief propagation on pairwise MRF, we apply belief propagation on factor graph to optimize terms which cannot be represented by pairwise restricted belief propagation. We propose a novel approximate method to reduce the computation complexity, and this approximate method suggests a direction for using belief propagation on factor graph to optimize high order potential functions similar to ours.

A computational framework to empower probabilistic protein design.

Motivation: The task of engineering a protein to perform a target biological function is known as protein design. A commonly used paradigm casts this functional design problem as a structural one, assuming a fixed backbone. In probabilistic protein design, positional amino acid probabilities are used to create a random library of sequences to be simultaneously screened for biological activity. Clearly, certain choices of probability distributions will be more successful in yielding functional sequences. However, since the number of sequences is exponential in protein length, computational optimization of the distribution is difficult.Results: In this paper, we develop a computational framework for probabilistic protein design following the structural paradigm. We formulate the distribution of sequences for a structure using the Boltzmann distribution over their free energies. The corresponding probabilistic graphical model is constructed, and we apply belief propagation (BP) to calculate marginal amino acid probabilities. We test this method on a large structural dataset and demonstrate the superiority of BP over previous methods. Nevertheless, since the results obtained by BP are far from optimal, we thoroughly assess the paradigm using high-quality experimental data. We demonstrate that, for small scale sub-problems, BP attains identical results to those produced by exact inference on the paradigmatic model. However, quantitative analysis shows that the distributions predicted significantly differ from the experimental data. These findings, along with the excellent performance we observed using BP on the smaller problems, suggest potential shortcomings of the paradigm. We conclude with a discussion of how it may be improved in the future.Contact: fromer@cs.huji.ac.il

A new look at survey propagation and its generalizations

This article provides a new conceptual perspective on survey propagation , which is an iterative algorithm recently introduced by the statistical physics community that is very effective in solving random k -SAT problems even with densities close to the satisfiability threshold. We first describe how any SAT formula can be associated with a novel family of Markov random fields (MRFs), parameterized by a real number ρ ∈ [0, 1]. We then show that applying belief propagation---a well-known “message-passing” technique for estimating marginal probabilities---to this family of MRFs recovers a known family of algorithms, ranging from pure survey propagation at one extreme (ρ = 1) to standard belief propagation on the uniform distribution over SAT assignments at the other extreme (ρ = 0). Configurations in these MRFs have a natural interpretation as partial satisfiability assignments, on which a partial order can be defined. We isolate cores as minimal elements in this partial ordering, which are also fixed points of survey propagation and the only assignments with positive probability in the MRF for ρ = 1. Our experimental results for k = 3 suggest that solutions of random formulas typically do not possess non-trivial cores. This makes it necessary to study the structure of the space of partial assignments for ρ < 1 and investigate the role of assignments that are very close to being cores. To that end, we investigate the associated lattice structure, and prove a weight-preserving identity that shows how any MRF with ρ > 0 can be viewed as a “smoothed” version of the uniform distribution over satisfying assignments (ρ = 0). Finally, we isolate properties of Gibbs sampling and message-passing algorithms that are typical for an ensemble of k -SAT problems.

Decoding LDPC Codes Over Integer Residue Rings

This correspondence presents a multistage decoding approach for a free Zopfq-submodule of Zopfq N of rank K defined by a sparse (N-K)timesN parity-check matrix overZopf q where q=pm, p=2 and m>1. The proposed method involves the repeated application of belief propagation decoding to exploit the natural ring epimorphism ZopfqrarrZopfp l:r|rarr Sigma i=0 l-1r(i)pi with kernel p lZopfq for each l, 1lesllesm, where Sigma i=0 m-1r(i)pi is the p-adic expansion of r. Computer simulations for codes of rate half and moderate length on an additive white Gaussian noise (AWGN) channel with various modulation schemes show that such a decoding strategy offers an additional coding gain of between 0.07-0.1 dB over a single-stage decoding approach

Community detection as an inference problem

We express community detection as an inference problem of determining the most likely arrangement of communities. We then apply belief propagation and mean-field theory to this problem, and show that this leads to fast, accurate algorithms for community detection.

Belief Propagation Decoding of Reed-Solomon Codes; a Bit-Level Soft Decision Decoding Algorithm

In this article we propose the application of Belief Propagation (BP) algorithm as a novel bit-level soft decision decoding (SDD) technique for Reed-Solomon (RS) codes. A brief tutorial on Belief Propagation algorithm is presented. A central issue in the application of BP algorithm to decoding RS codes is the construction of a sparse parity check matrix for the binary image of the code. It is demonstrated that Vardy's technique may be applied to find a sparse parity check matrix for RS codes. However, this technique is not applicable to all cases. The BP algorithm is applied to two test codes. In one case, simulation models show that the BP algorithm outperforms the hard decision Euclidean decoding by more than 2 dB of additional coding gain. The results with the second test code are not as promising.