Minimum Probability Flow Research Articles

Inferring Cultural Landscapes with the Inverse Ising Model.

The space of possible human cultures is vast, but some cultural configurations are more consistent with cognitive and social constraints than others. This leads to a "landscape" of possibilities that our species has explored over millennia of cultural evolution. However, what does this fitness landscape, which constrains and guides cultural evolution, look like? The machine-learning algorithms that can answer these questions are typically developed for large-scale datasets. Applications to the sparse, inconsistent, and incomplete data found in the historical record have received less attention, and standard recommendations can lead to bias against marginalized, under-studied, or minority cultures. We show how to adapt the minimum probability flow algorithm and the Inverse Ising model, a physics-inspired workhorse of machine learning, to the challenge. A series of natural extensions-including dynamical estimation of missing data, and cross-validation with regularization-enables reliable reconstruction of the underlying constraints. We demonstrate our methods on a curated subset of the Database of Religious History: records from 407 religious groups throughout human history, ranging from the Bronze Age to the present day. This reveals a complex, rugged, landscape, with both sharp, well-defined peaks where state-endorsed religions tend to concentrate, and diffuse cultural floodplains where evangelical religions, non-state spiritual practices, and mystery religions can be found.

MPF–BML: a standalone GUI-based package for maximum entropy model inference

Learning underlying correlation patterns in data is a central problem across scientific fields. Maximum entropy models present an important class of statistical approaches for addressing this problem. However, accurately and efficiently inferring model parameters are a major challenge, particularly for modern high-dimensional applications such as in biology, for which the number of parameters is enormous. Previously, we developed a statistical method, minimum probability flow-Boltzmann Machine Learning (MPF-BML), for performing fast and accurate inference of maximum entropy model parameters, which was applied to genetic sequence data to estimate the fitness landscape for the surface proteins of human immunodeficiency virus and hepatitis C virus. To facilitate seamless use of MPF-BML and encourage more widespread application to data in diverse fields, we present a standalone cross-platform package of MPF-BML which features an easy-to-use graphical user interface. The package only requires the input data (protein sequence data or data of multiple configurations of a complex system with large number of variables) and returns the maximum entropy model parameters. The MPF-BML software is publicly available under the MIT License at https://github.com/ahmedaq/MPF-BML-GUI. Supplementary data are available at Bioinformatics online.

Convenient Interface to Inverse Ising (ConIII): A Python 3 Package for Solving Ising-Type Maximum Entropy Models

ConIII (pronounced CON-ee) is an open-source Python project providing a simple interface to solving the pairwise and higher order Ising model and a base for extension to other maximum entropy models. We describe the maximum entropy problem and give an overview of the algorithms that are implemented as part of ConIII ( https://github.com/eltrompetero/coniii ) including Monte Carlo histogram, pseudolikelihood, minimum probability flow, a regularized mean field method, and a cluster expansion method. Our goal is to make a variety of maximum entropy techniques accessible to those unfamiliar with the techniques and accelerate workflow for users. Funding Statement: EDL was supported by an NSF Graduate Fellowship under grant no. DGE-1650441. This research was supported in part by a congressional research grant provided by the Dirksen Congressional Center.

Variational Probability Flow for Biologically Plausible Training of Deep Neural Networks

The quest for biologically plausible deep learning is driven, not just by the desire to explain experimentally-observed properties of biological neural networks, but also by the hope of discovering more efficient methods for training artificial networks. In this paper, we propose a new algorithm named Variational Probably Flow (VPF), an extension of minimum probability flow for training binary Deep Boltzmann Machines (DBMs). We show that weight updates in VPF are local, depending only on the states and firing rates of the adjacent neurons. Unlike contrastive divergence, there is no need for Gibbs confabulations; and unlike backpropagation, alternating feedforward and feedback phases are not required. Moreover, the learning algorithm is effective for training DBMs with intra-layer connections between the hidden nodes. Experiments with MNIST and Fashion MNIST demonstrate that VPF learns reasonable features quickly, reconstructs corrupted images more accurately, and generates samples with a high estimated log-likelihood. Lastly, we note that, interestingly, if an asymmetric version of VPF exists, the weight updates directly explain experimental results in Spike-Timing-Dependent Plasticity (STDP).

Learning Algorithm of Boltzmann Machine Based on Spatial Monte Carlo Integration Method

The machine learning techniques for Markov random fields are fundamental in various fields involving pattern recognition, image processing, sparse modeling, and earth science, and a Boltzmann machine is one of the most important models in Markov random fields. However, the inference and learning problems in the Boltzmann machine are NP-hard. The investigation of an effective learning algorithm for the Boltzmann machine is one of the most important challenges in the field of statistical machine learning. In this paper, we study Boltzmann machine learning based on the (first-order) spatial Monte Carlo integration method, referred to as the 1-SMCI learning method, which was proposed in the author’s previous paper. In the first part of this paper, we compare the method with the maximum pseudo-likelihood estimation (MPLE) method using a theoretical and a numerical approaches, and show the 1-SMCI learning method is more effective than the MPLE. In the latter part, we compare the 1-SMCI learning method with other effective methods, ratio matching and minimum probability flow, using a numerical experiment, and show the 1-SMCI learning method outperforms them.

Parameter inference in a probabilistic model from data: Regulation of transition rate in the Monte Carlo method

We consider the inference of parameters in a probabilistic model from a data set, which is generated by an unknown probabilistic model. The Monte Carlo method is a tool for obtaining a data set obeying a given probability distribution. A set of transition rates is required to satisfy three conditions (irreducible, aperiodic, and stationary) for a sampled data set to represent a probability distribution. We utilize the stationary condition of a probability distribution with respect to transition rates to infer parameters. A frequency distribution by a data set substitutes for an unknown probability distribution in the condition. Our method includes minimum probability flow as a special case and becomes superior to it as the number of samples increases.

Robust Discovery of Temporal Structure in Multi-neuron Recordings Using Hopfield Networks

Abstract We present here a novel method for the classical task of extracting reoccurring spatiotemporal patterns from spiking activity of large populations of neurons. In contrast to previous studies that mainly focus on synchrony detection or exactly recurring binary patterns, we perform the search in an approximate way that clusters together nearby, noisy network states in the data. Our approach is to use minimum probability flow (MPF) parameter estimation to determinis- tically fit very large Hopfield networks on windowed spike trains obtained from recordings of spontaneous activity of neurons in cat visual cortex. Examining the structure of the network memories over the spiking activity after training, we find that the networks robustly discover long-range temporal correlations. Specifically, the recurrent network dynamics denoise and group together windowed spike patterns, revealing underlying structure such as cell assemblies. We first demonstrate this by computing an analogy to spike triggered averages that we call memory triggered averages (MTAs). MTAs are obtained by averaging raw spike train windows that converge under the network dynamics to the same memory. The MTAs reveal promi- nent repeating patterns in the data that are difficult to detect with standard methods such as PCA. Additionally, when memories are collected over eight disjoint epochs in 280 seconds of windowed spiking activity from 50 neurons, their counts are nearly identical and the networks store significantly more memories than would be possible if trained on random patterns.

Modeling Higher-Order Correlations within Cortical Microcolumns

We statistically characterize the population spiking activity obtained from simultaneous recordings of neurons across all layers of a cortical microcolumn. Three types of models are compared: an Ising model which captures pairwise correlations between units, a Restricted Boltzmann Machine (RBM) which allows for modeling of higher-order correlations, and a semi-Restricted Boltzmann Machine which is a combination of Ising and RBM models. Model parameters were estimated in a fast and efficient manner using minimum probability flow, and log likelihoods were compared using annealed importance sampling. The higher-order models reveal localized activity patterns which reflect the laminar organization of neurons within a cortical column. The higher-order models also outperformed the Ising model in log-likelihood: On populations of 20 cells, the RBM had 10% higher log-likelihood (relative to an independent model) than a pairwise model, increasing to 45% gain in a larger network with 100 spatiotemporal elements, consisting of 10 neurons over 10 time steps. We further removed the need to model stimulus-induced correlations by incorporating a peri-stimulus time histogram term, in which case the higher order models continued to perform best. These results demonstrate the importance of higher-order interactions to describe the structure of correlated activity in cortical networks. Boltzmann Machines with hidden units provide a succinct and effective way to capture these dependencies without increasing the difficulty of model estimation and evaluation.

Novel local learning rule for neural adaptation fits Hopfield memory networks efficiently and optimally

We present an algorithm to store binary memories in a Little-Hopfield neural network using minimum probability flow, a recent technique to fit parameters in energy-based probabilistic models. For memories without noise, our algorithm provably achieves optimal pattern storage and outperforms classical methods both in speed and memory recovery. Moreover, when trained on noisy or corrupted versions of a fixed set of binary patterns, our algorithm finds networks which correctly store the originals. We also demonstrate this finding visually with the unsupervised storage and clean-up of large binary fingerprint images from significantly corrupted samples.

New Method for Parameter Estimation in Probabilistic Models: Minimum Probability Flow

Fitting probabilistic models to data is often difficult, due to the general intractability of the partition function. We propose a new parameter fitting method, minimum probability flow (MPF), which is applicable to any parametric model. We demonstrate parameter estimation using MPF in two cases: a continuous state space model, and an Ising spin glass. In the latter case, MPF outperforms current techniques by at least an order of magnitude in convergence time with lower error in the recovered coupling parameters.

The Ising decoder: reading out the activity of large neural ensembles

The Ising model has recently received much attention for the statistical description of neural spike train data. In this paper, we propose and demonstrate its use for building decoders capable of predicting, on a millisecond timescale, the stimulus represented by a pattern of neural activity. After fitting to a training dataset, the Ising decoder can be applied "online" for instantaneous decoding of test data. While such models can be fit exactly using Boltzmann learning, this approach rapidly becomes computationally intractable as neural ensemble size increases. We show that several approaches, including the Thouless-Anderson-Palmer (TAP) mean field approach from statistical physics, and the recently developed Minimum Probability Flow Learning (MPFL) algorithm, can be used for rapid inference of model parameters in large-scale neural ensembles. Use of the Ising model for decoding, unlike other problems such as functional connectivity estimation, requires estimation of the partition function. As this involves summation over all possible responses, this step can be limiting. Mean field approaches avoid this problem by providing an analytical expression for the partition function. We demonstrate these decoding techniques by applying them to simulated neural ensemble responses from a mouse visual cortex model, finding an improvement in decoder performance for a model with heterogeneous as opposed to homogeneous neural tuning and response properties. Our results demonstrate the practicality of using the Ising model to read out, or decode, spatial patterns of activity comprised of many hundreds of neurons.