Message-passing Methods Research Articles

Quantum Annealing for Computer Vision minimization problems

Computer Vision (CV) labeling problems play a pivotal role in low-level vision. For decades, it has been known that these problems can be elegantly formulated as discrete energy-minimization problems derived from probabilistic graphical models such as Markov Random Fields (MRFs). Despite recent advances in MRF inference algorithms (such as graph-cut and message-passing methods), the resulting energy-minimization problems are generally viewed as intractable. The emergence of quantum computations, which offer the potential for faster solutions to certain problems than classical methods, has led to an increased interest in utilizing quantum properties to overcome intractable problems. Recently, there has also been a growing interest in Quantum Computer Vision (QCV), hoping to provide a credible alternative/assistant to deep learning solutions. This study investigates a new Quantum Annealing-based inference algorithm for CV discrete energy minimization problems. Our contribution is focused on Stereo Matching as a significant CV labeling problem. As a proof of concept, we also use a hybrid quantum–classical solver provided by D-Wave System to compare our results with the best classical inference algorithms in the literature. Our results show that Quantum Annealing can yield promising results for Stereo Matching problems, with improved accuracy on certain stereo images and competitive performance on others.

Tensor Network Message Passing.

When studying interacting systems, computing their statistical properties is a fundamental problem in various fields such as physics, applied mathematics, and machine learning. However, this task can be quite challenging due to the exponential growth of the state space as the system size increases. Many standard methods have significant weaknesses. For instance, message-passing algorithms can be inaccurate and even fail to converge due to short loops, while tensor network methods can have exponential computational complexity in large graphs due to long loops. In this Letter, we propose a new method called "tensor network message passing." This approach allows us to compute local observables like marginal probabilities and correlations by combining the strengths of tensor networks in contracting small subgraphs with many short loops and the strengths of message-passing methods in globally sparse graphs, thus addressing the crucial weaknesses of both approaches. Our algorithm is exact for systems that are globally treelike and locally dense-connected when the dense local graphs have a limited tree width. We have conducted numerical experiments on synthetic and real-world graphs to compute magnetizations of Ising models and spin glasses, and have demonstrated the superiority of our approach over standard belief propagation and the recently proposed loopy message-passing algorithm. In addition, we discuss the potential applications of our method in inference problems in networks, combinatorial optimization problems, and decoding problems in quantum error correction.

Message passing on networks with loops

Message passing is a fundamental technique for performing calculations on networks and graphs with applications in physics, computer science, statistics, and machine learning, including Bayesian inference, spin models, satisfiability, graph partitioning, network epidemiology, and the calculation of matrix eigenvalues. Despite its wide use, however, it has long been recognized that the method has a fundamental flaw: It works poorly on networks that contain short loops. Loops introduce correlations that can cause the method to give inaccurate answers or to fail completely in the worst cases. Unfortunately, most real-world networks contain many short loops, which limits the usefulness of the message-passing approach. In this paper we demonstrate how to rectify this shortcoming and create message-passing methods that work on any network. We give 2 example applications, one to the percolation properties of networks and the other to the calculation of the spectra of sparse matrices.

Integrated Cooperative Synchronization for Wireless Sensor Networks

A precise decentralized clock synchronization method, referred to as integrated cooperative synchronization (ICS) is proposed in this letter. ICS is a delay compensating method that relies on mean-field message passing, where the measurement phase is integrated into the message passing phase. By selecting an extended factorization of the underlying joint a-posteriori distribution of the clock parameters and an appropriate message scheduling for link initialization, ICS is conceptually much simpler than conventional message-passing methods while achieving similar accuracy and reduced computational complexity. Moreover, it has only slightly higher implementation requirements compared to less accurate consensus methods.

Random Sampling with Interspike-Intervals of the Exponential Integrate and Fire Neuron: A Computational Interpretation of UP-States.

Oscillations between high and low values of the membrane potential (UP and DOWN states respectively) are an ubiquitous feature of cortical neurons during slow wave sleep and anesthesia. Nevertheless, a surprisingly small number of quantitative studies have been conducted only that deal with this phenomenon’s implications for computation.Here we present a novel theory that explains on a detailed mathematical level the computational benefits of UP states. The theory is based on random sampling by means of interspike intervals (ISIs) of the exponential integrate and fire (EIF) model neuron, such that each spike is considered a sample, whose analog value corresponds to the spike’s preceding ISI. As we show, the EIF’s exponential sodium current, that kicks in when balancing a noisy membrane potential around values close to the firing threshold, leads to a particularly simple, approximative relationship between the neuron’s ISI distribution and input current. Approximation quality depends on the frequency spectrum of the current and is improved upon increasing the voltage baseline towards threshold. Thus, the conceptually simpler leaky integrate and fire neuron that is missing such an additional current boost performs consistently worse than the EIF and does not improve when voltage baseline is increased. For the EIF in contrast, the presented mechanism is particularly effective in the high-conductance regime, which is a hallmark feature of UP-states.Our theoretical results are confirmed by accompanying simulations, which were conducted for input currents of varying spectral composition. Moreover, we provide analytical estimations of the range of ISI distributions the EIF neuron can sample from at a given approximation level. Such samples may be considered by any algorithmic procedure that is based on random sampling, such as Markov Chain Monte Carlo or message-passing methods.Finally, we explain how spike-based random sampling relates to existing computational theories about UP states during slow wave sleep and present possible extensions of the model in the context of spike-frequency adaptation.

MRF Energy Minimization and Beyond via Dual Decomposition

This paper introduces a new rigorous theoretical framework to address discrete MRF-based optimization in computer vision. Such a framework exploits the powerful technique of Dual Decomposition. It is based on a projected subgradient scheme that attempts to solve an MRF optimization problem by first decomposing it into a set of appropriately chosen subproblems, and then combining their solutions in a principled way. In order to determine the limits of this method, we analyze the conditions that these subproblems have to satisfy and demonstrate the extreme generality and flexibility of such an approach. We thus show that by appropriately choosing what subproblems to use, one can design novel and very powerful MRF optimization algorithms. For instance, in this manner we are able to derive algorithms that: 1) generalize and extend state-of-the-art message-passing methods, 2) optimize very tight LP-relaxations to MRF optimization, and 3) take full advantage of the special structure that may exist in particular MRFs, allowing the use of efficient inference techniques such as, e.g., graph-cut-based methods. Theoretical analysis on the bounds related with the different algorithms derived from our framework and experimental results/comparisons using synthetic and real data for a variety of tasks in computer vision demonstrate the extreme potentials of our approach.

Mutable objects in R

Programming paradigms help us understand the differences and similarities between fundamental choices in language design. This paper looks at R in the context of three paradigms of object oriented programming: mutable versus immutable objects, generic-function versus message-passing methods, and class-based versus prototype-based inheritance. The paper also describes a new OO package for R, mutatr, which provides mutable objects with message-passing methods and prototype-based inheritance. The mutatr package is available on CRAN.

Web-based Distributed Network Analyzer using a System Entity Structure over a Service-oriented Architecture

As a network’s uses, and especially the number of internet users, increases rapidly, an efficient system for managing large-network traffic datasets becomes an important issue. Although there are several network traffic analysis tools, such as tcpdump, Ethereal, and other applications, these tools have weaknesses, namely the limited size of files, the use of command line execution, the large memory and huge computational power requirements. In addition to these scalability limitations, both tcpdump and Ethereal have security issues. Files captured by these tools keep all of the packet information, such as internet protocol (IP) addresses, port numbers, and packet sizes. As well as basic network traffic information, the captured files contain secure information: user identification numbers (IDs) and passwords. Therefore, the captured files should not be allowed to be leaked out. However, network analyses need to be performed outside the target networks in some cases. This paper presents an approach to efficiently and quickly analyze a large number of network behaviors. This is achieved by applying System Entity Structure (SES) theory. To speed up evaluation time, a web-based distributed simulation approach over Service-oriented Architecture (SOA) is applied. Discrete Event System Specification/Service-oriented Architecture (DEVS/SOA) is used to deploy workloads into multi-servers, increasing overall system performance. A web-based distributed simulation contains two fundamental processes: distributing and analyzing among loosely coupled models through message-passing methods. The distributed simulation — allocating distributing models inside networks and assigning analyzing models outside networks — also allows the analysis of network behaviors out of networks while keeping important information secured.

Passing Messages to Lonely Numbers

Message-passing methods provide powerful approximation algorithms for problems that can be formulated in terms of (probabilistic) graphical models. These methods find applications in statistical physics, inference, and combinatorial optimization. Sudoku, a popular number puzzle, is a simple optimization problem that message-passing algorithms can help solve. Therefore, Sudoku is an ideal vehicle to demonstrate these methods' strengths and limitations.

Estimating the Wrong Graphical Model: Benefits in the Computation-Limited Setting

Consider the problem of joint parameter estimation and prediction in a Markov random field: that is, the model parameters are estimated on the basis of an initial set of data, and then the fitted model is used to perform prediction (e.g., smoothing, denoising, interpolation) on a new noisy observation. Working under the restriction of limited computation, we analyze a joint method in which the same convex variational relaxation is used to construct an M-estimator for fitting parameters, and to perform approximate marginalization for the prediction step. The key result of this paper is that in the computation-limited setting, using an inconsistent parameter estimator (i.e., an estimator that returns the wrong model even in the infinite data limit) is provably beneficial, since the resulting errors can partially compensate for errors made by using an approximate prediction technique. En route to this result, we analyze the asymptotic properties of M-estimators based on convex variational relaxations, and establish a Lipschitz stability property that holds for a broad class of convex variational methods. This stability result provides additional incentive, apart from the obvious benefit of unique global optima, for using message-passing methods based on convex variational relaxations. We show that joint estimation/prediction based on the reweighted sum-product algorithm substantially outperforms a commonly used heuristic based on ordinary sum-product.

Parallel finite element calculation of flow in a three-dimensional lid-driven cavity using the CM-5 and T3D

Steady flows in a three-dimensional lid-driven cavity at moderate Reynolds number are studied using various methods of parallel programming on the Cray T3D and Thinking Machines CM-5. These three-dimensional flows are compared with flows computed in a two-dimensional cavity. Solutions at Reynolds number up to 500 agree well with the experimental data of Aidun et al. (Phys. Fluids A, 3, 2081–2091 (1991)) for the location of separation of the secondary eddy at the downstream wall. Convergence of the three-dimensional problem using GMRES with diagonal preconditioning could not be obtained at Reynolds number greater than about 500. We speculate that the source of the difficulty is the loss of stability via pitchfork and Hopf bifurcations identified by Aidun et al. The relative performance of various methods of message passing on the Cray T3D is compared with the data-parallel mode of programming on the CM-5. No clear advantage between machines or message-passing methods is distinguished. © 1997 John Wiley & Sons, Ltd.

A new method for efficient symbolic propagation in discrete Bayesian networks

This paper presents a new efficient method for uncertainty propagation in discrete Bayesian networks in symbolic, as opposed to numeric, form, when considering some of the probabilities of the Bayesian network as parameters. The algebraic structure of the conditional probabilities of any set of nodes, given some evidence, is characterized as ratios of linear polynomials in the parameters. We use this result to carry out these symbolic expressions efficiently by calculating the coefficients of the polynomials involved, using standard numerical algorithms. The numeric canonical components method is proposed as an alternative to symbolic computations, gaining in speed and simplicity. It is also shown how to avoid redundancy when calculating the numeric canonical components probabilities using standard messagepassing methods. The canonical components can also be used to obtain lower and upper bounds for the symbolic expression associated with the probabilities. Finally, we analyze the problem of symbolic evidence, which allows answering multiple queries regarding a given set of evidential nodes. In this case, the algebraic structure of the symbolic expressions obtained for the probabilities are shown to be ratios of nonlinear polynomial expressions. Then, we can perform symbolic inference with only a small set of symbolic evidential nodes. The methodology is illustrated by examples.

A new method for efficient symbolic propagation in discrete Bayesian networks

This paper presents a new efficient method for uncertainty propagation in discrete Bayesian networks in symbolic, as opposed to numeric, form, when considering some of the probabilities of the Bayesian network as parameters. The algebraic structure of the conditional probabilities of any set of nodes, given some evidence, is characterized as ratios of linear polynomials in the parameters. We use this result to carry out these symbolic expressions efficiently by calculating the coefficients of the polynomials involved, using standard numerical algorithms. The numeric canonical components method is proposed as an alternative to symbolic computations, gaining in speed and simplicity. It is also shown how to avoid redundancy when calculating the numeric canonical components probabilities using standard message-passing methods. The canonical components can also be used to obtain lower and upper bounds for the symbolic expression associated with the probabilities. Finally, we analyze the problem of symbolic evidence, which allows answering multiple queries regarding a given set of evidential nodes. In this case, the algebraic structure of the symbolic expressions obtained for the probabilities are shown to be ratios of nonlinear polynomial expressions. Then, we can perform symbolic inference with only a small set of symbolic evidential nodes. The methodology is illustrated by examples. © 1996 John Wiley & Sons, Inc.