Inference Algorithms For Models Research Articles

Community Detection on Social Networks With Sentimental Interaction

Many studies on community detection are mainly based on the similarity in friendship between users. Recent studies have started to explore node contents to identify semantically meaningful communities. However, the sentimental interaction information which plays an important role in community detection is often ignored. By analyzing and utilizing the abundant sentimental interaction information, one can not only more precisely identify the communities, but also discover the interesting interactions and conflicts between these communities. Based on this concept, the authors propose a new Community Sentiment Diffusion Detection Model (CSDD), which utilizes sentimental information embedded in forward posts. Furthermore, the authors present an efficient variational algorithm for model inference. The community detection results have been verified on two large Twitter datasets. It is experimentally demonstrated that we can provide a fine-grained view of sentimental interaction between communities and discover the mechanism of sentiment diffusion between communities.

Learning-Based Testing Using SAL (Symbolic Analysis Laboratory) Model Checker

This paper studies learning-based testing (LBT) for reactive systems with different learning algorithms and model checkers. LBT is a technique that requires a learning algorithm to learn the models to generate test cases automatically. We have used the generic methodology of LBT to test reactive systems with two different model inference algorithms (i.e., IKL, DKL) and two different model checking tools (i.e., NuSMV, SAL). To investigate the feasibility of LBT, we integrated our SUTs with these algorithms in LBT and tested if LBT optimizes test generation with these algorithms. We tested our SUTs with Boolean data types to check the difference in the working of model inference and model checking algorithms which we analyzed experimentally. The results show that LBT works better with DKL and SAL. DKL is a recently proposed model inference algorithm, and SAL is the latest model checker on which the research is being carried out. DKL and SAL algorithms explore errors in reactive SUTs with the h LBT framework more quickly and efficiently.

PyVBMC: Efficient Bayesian inference in Python

PyVBMC is a Python implementation of the Variational Bayesian Monte Carlo (VBMC) algorithm for posterior and model inference for black-box computational models (Acerbi, 2018, 2020). VBMC is an approximate inference method designed for efficient parameter estimation and model assessment when model evaluations are mildly-to-very expensive (e.g., a second or more) and/or noisy. Specifically, VBMC computes: - a flexible (non-Gaussian) approximate posterior distribution of the model parameters, from which statistics and posterior samples can be easily extracted; - an approximation of the model evidence or marginal likelihood, a metric used for Bayesian model selection. PyVBMC can be applied to any computational or statistical model with up to roughly 10-15 continuous parameters, with the only requirement that the user can provide a Python function that computes the target log likelihood of the model, or an approximation thereof (e.g., an estimate of the likelihood obtained via simulation or Monte Carlo methods). PyVBMC is particularly effective when the model takes more than about a second per evaluation, with dramatic speed-ups of 1-2 orders of magnitude when compared to traditional approximate inference methods. Extensive benchmarks on both artificial test problems and a large number of real models from the computational sciences, particularly computational and cognitive neuroscience, show that VBMC generally - and often vastly - outperforms alternative methods for sample-efficient Bayesian inference, and is applicable to both exact and simulator-based models (Acerbi, 2018, 2019, 2020). PyVBMC brings this state-of-the-art inference algorithm to Python, along with an easy-to-use Pythonic interface for running the algorithm and manipulating and visualizing its results.

Bayesian Kernelized Matrix Factorization for Spatiotemporal Traffic Data Imputation and Kriging

Missingness and corruption are common problems for real-world traffic data. How to accurately perform imputation and prediction based on incomplete or even sparse traffic data becomes a critical research question in intelligent transportation systems. Low-rank matrix factorization (MF) is a common solution for the general missing value imputation problem. To better characterize and encode the strong spatial and temporal consistency in traffic data, existing work has introduced flexible spatial/temporal Gaussian process (GP) priors to model the latent factors in MF framework, which also allows us to perform kriging for unseen locations and virtual sensors. However, learning the hyperparameters in GP kernels remains a challenging task. In this paper, we present a Bayesian kernelized matrix factorization (BKMF) model with an efficient Markov chain Monte Carlo (MCMC) sampling algorithm for model inference. By learning the kernel hyperparameters from their marginal posteriors through a slice sampling treatment and updating the latent factors alternatively with Gibbs sampling, we achieve a fully Bayesian model for the spatiotemporally kernelized (i.e., GP prior regularized) MF framework. We apply BKMF on both imputation and kriging tasks, and our results demonstrate the superiority of BKMF compared with state-of-the-art spatiotemporal models. In addition, we also explore the effects of different GP kernels in characterizing networked spatiotemporal traffic state data.

Data-Driven-Based Stochastic Robust Optimization for a Virtual Power Plant With Multiple Uncertainties

Virtual power plant (VPP) has gradually become a key technology to the increasing penetration of renewable energy. The uncertainty and variability of renewable energy and load demand pose significant challenges to the efficiency and stability of VPP's operation. In this paper, a data-driven two-stage stochastic robust optimization (SRO) scheduling model is proposed for a VPP considering wind power, solar power, load demand, and market price uncertainties. The objective is to maximize the profit of the VPP in the energy market and minimize the total system cost of the VPP in the reserve market under the worst-case realization of the uncertainties. The Dirichlet process mixture model (DPMM) and variational inference algorithm are employed for constructing the data-driven uncertainty ambiguity set considering the correlations among multiple uncertainties. The tailored column-and-constraint generation algorithm is developed to solve the SRO model iteratively by reformulating the second stage with the application of the Karush-Kuhn-Tucker conditions. Results from a case study illustrate the effectiveness and superiority of the proposed model.

Robust Detection of Link Communities With Summary Description in Social Networks

Community detection has been extensively studied for various applications. Recent research has started to explore node contents to identify semantically meaningful communities. However, links in real networks typically have semantic descriptions and communities of links can better characterize community behaviors than communities of nodes. The second issue in community finding is that the most existing methods assume network topologies and descriptive contents carry the same or compatible information of node group membership, restricting them to one topic per community, which is generally violated in real networks. The third issue is that the existing methods use top ranked words or phrases to label topics when interpreting communities, which is often inadequate for comprehension. To address these issues altogether, we propose a new Bayesian probabilistic approach for modeling real networks and developing an efficient variational algorithm for model inference. Our new method explores the intrinsic correlation between communities and topics to discover link communities and extract semantically meaningful community summaries at the same time. If desired, it is able to derive more than one topical summary per community to provide rich explanations. We present experimental results to show the effectiveness of our new approach and evaluate the method by a case study.

Leveraging Cognitive Diagnosis to Improve Peer Assessment in MOOCs

A major challenge faced by popular massive open online courses (MOOCs) is the assessment of large-scale open-ended assignments submitted by students. Recently, peer assessment has become a mainstream paradigm that helps grade open-ended assignments on a large scale. In peer assessment, students also become graders in grading a small number of their peers' assignments, and the peer grades are then aggregated to predict a true score for each assignment. The collected peer grades are usually inaccurate because graders have different reliabilities and biases. To improve accuracy, several probabilistic graph models have been proposed to model the reliability and bias of each grader. However, none of these models consider graders' competency information in the assignments to be graded, which has been found to be very effective. We propose two new probabilistic graph models to improve the accuracy of cardinal peer assessments based on the well-accepted cognitive diagnosis technique. Specifically, the cognitive diagnosis model DINA is applied to determine grader competency based on historical tests or assignments. Then, this information is used to optimize the modeling of grader reliability in each of the proposed models. Moreover, an effective model inference algorithm is proposed to infer true scores of assignments. Experimental results based on real world datasets show that the two proposed models outperform state-of-the-art models and that consideration of grader competency contributes to improved score estimation.

Sparse Regression in Cancer Genomics: Comparing Variable Selection and Predictions in Real World Data.

Background:Evaluation of gene interaction models in cancer genomics is challenging, as the true distribution is uncertain. Previous analyses have benchmarked models using synthetic data or databases of experimentally verified interactions – approaches which are susceptible to misrepresentation and incompleteness, respectively. The objectives of this analysis are to (1) provide a real-world data-driven approach for comparing performance of genomic model inference algorithms, (2) compare the performance of LASSO, elastic net, best-subset selection, penalisation and penalisation in real genomic data and (3) compare algorithmic preselection according to performance in our benchmark datasets to algorithmic selection by internal cross-validation.Methods:Five large genomic datasets were extracted from Gene Expression Omnibus. ‘Gold-standard’ regression models were trained on subspaces of these datasets (, ). Penalised regression models were trained on small samples from these subspaces () and validated against the gold-standard models. Variable selection performance and out-of-sample prediction were assessed. Penalty ‘preselection’ according to test performance in the other 4 datasets was compared to selection internal cross-validation error minimisation.Results:-penalisation achieved the highest cosine similarity between estimated coefficients and those of gold-standard models. -penalised models explained the greatest proportion of variance in test responses, though performance was unreliable in low signal:noise conditions. also attained the highest overall median variable selection F1 score. Penalty preselection significantly outperformed selection by internal cross-validation in each of 3 examined metrics.Conclusions:This analysis explores a novel approach for comparisons of model selection approaches in real genomic data from 5 cancers. Our benchmarking datasets have been made publicly available for use in future research. Our findings support the use of penalisation for structural selection and penalisation for coefficient recovery in genomic data. Evaluation of learning algorithms according to observed test performance in external genomic datasets yields valuable insights into actual test performance, providing a data-driven complement to internal cross-validation in genomic regression tasks.

Detecting Communities with Multiplex Semantics by Distinguishing Background, General, and Specialized Topics

Finding semantic communities using network topology and contents together is a hot topic in community detection. Existing methods often use word attributes in an indiscriminate way to help finding communities. Through analysis we find that, words in networked contents often embody a hierarchical semantic structure. Some words reflect a background topic of the whole network with all communities, some imply the high-level general topic covering several topic-related communities, and some imply the high-resolution specialized topic to describe each community. Ignoring such semantic structures often leads to defects in depicting networked contents where deep semantics are not fully utilized. To solve this problem, we propose a new Bayesian probabilistic model. By distinguishing words from either a background topic or some two-level topics (i.e., general and specialized topics), this model not only better utilizes the networked contents to help finding communities, but also provides a clearer multiplex semantic community interpretation. We then give an efficient variational algorithm for model inference. The superiority of this new approach is demonstrated by comparing with ten state-of-the-art methods on nine real networks and an artificial benchmark. A case study is further provided to show its strong ability in deep semantic interpretation of communities.

Identification of Generalized Semantic Communities in Large Social Networks

Community detection in networks is a fundamental data analysis task. Recently, researchers have tried to improve its performance by exploiting semantic contents and interpret the communities. However, they typically assume that communities are assortative (i.e. vertices are mostly connected to others within the group), thus they cannot find the generalized community structures, which includes assortative communities, disassortative communities (i.e. most connections are from two groups), or a combination. In addition, they often assume that each group membership corresponds to a single topic, thus they cannot perform well when the contents are not consistent with community structures. To address these two issues, we propose a new Bayesian model and develop an efficient variational inference algorithm for model inference. This model describes the generalized communities and the topical clusters separately, and explores their latent correlation simultaneously to make the two parts mutually reinforcing. Our model is not only robust to the above problems, but also can interpret each community using more than one topic. We validate the robustness of this approach on an artificial benchmark, and analyze its interpretability by a case study. We finally show its superior community detection performance by comparing with eight state-of-the-art algorithms on eight real networks.

Spectral Mixture Model Inspired Network Architectures for Hyperspectral Unmixing

In many statistical hyperspectral unmixing approaches, the unmixing task is essentially an optimization problem given a defined linear or nonlinear spectral mixture model. However, most of the model inference algorithms require a time-consuming iterative procedure. On the other hand, neural networks have been recently used to estimate abundances given some training samples, or directly estimate endmembers and abundances simultaneously in an unsupervised setting. However, their disadvantages are clear: lack of interpretability and reliance on the large training set. Model-inspired neural networks are constructed by the problem model and its corresponding inference algorithm. It incorporates the prior knowledge of physical model and algorithm into network architecture, combining the advantages of model-based and learning-based methods. This article deeply unfolds the linear mixture model and the corresponding iterative shrinkage-thresholding algorithm (ISTA) to build two unmixing network architectures. The first assumes that the set of endmembers are known, and the deep unfolded ISTA model is only for abundance estimation; and the second is used for blind unmixing to estimate both endmembers and abundances at the same time. The networks can be trained by supervised and unsupervised schemes, respectively, with a small-size training set, and then, unmixing becomes a feedforward process, which is very fast since no iteration is required. The experimental results show their competitive performance compared with the state-of-the-art unmixing approaches.

An ultra-sensitive T-cell receptor detection method for TCR-Seq and RNA-Seq data.

T-cell receptors (TCRs) function to recognize antigens and play vital roles in T-cell immunology. Surveying TCR repertoires by characterizing complementarity-determining region 3 (CDR3) is a key issue. Due to the high diversity of CDR3 and technological limitation, accurate characterization of CDR3 repertoires remains a great challenge. We propose a computational method named CATT for ultra-sensitive and precise TCR CDR3 sequences detection. CATT can be applied on TCR sequencing, RNA-Seq and single-cell TCR(RNA)-Seq data to characterize CDR3 repertoires. CATT integrated de Bruijn graph-based micro-assembly algorithm, data-driven error correction model and Bayesian inference algorithm, to self-adaptively and ultra-sensitively characterize CDR3 repertoires with high performance. Benchmark results of datasets from in silico and experimental data demonstrated that CATT showed superior recall and precision compared with existing tools, especially for data with short read length and small size and single-cell sequencing data. Thus, CATT will be a useful tool for TCR analysis in researches of cancer and immunology. http://bioinfo.life.hust.edu.cn/CATT or https://github.com/GuoBioinfoLab/CATT. Supplementary data are available at Bioinformatics online.

Understanding the Dynamics of Message Passing Algorithms: A Free Probability Heuristics

We use freeness assumptions of random matrix theory to analyze the dynamical behavior of inference algorithms for probabilistic models with dense coupling matrices in the limit of large systems. For a toy Ising model, we are able to recover previous results such as the property of vanishing effective memories and the analytical convergence rate of the algorithm.

A dynamic network model with persistent links and node-specific latent variables, with an application to the interbank market

We propose a dynamic network model where two mechanisms control the probability of a link between two nodes: (i) the existence or absence of this link in the past, and (ii) node-specific latent variables (dynamic fitnesses) describing the propensity of each node to create links. Assuming a Markov dynamics for both mechanisms, we propose an Expectation-Maximization algorithm for model estimation and inference of the latent variables. The estimated parameters and fitnesses can be used to forecast the presence of a link in the future. We apply our methodology to the e-MID interbank network for which the two linkage mechanisms are associated with two different trading behaviors in the process of network formation, namely preferential trading and trading driven by node-specific characteristics. The empirical results allow to recognize preferential lending in the interbank market and indicate how a method that does not account for time-varying network topologies tends to overestimate preferential linkage.

Implementing Artificial Intelligence in Predicting Metrics for Characterizing Laser Propagation in Atmospheric Turbulence

The effects of a laser beam propagating through atmospheric turbulence are investigated using the phase screen approach. Turbulence effects are modeled by the Kolmogorov description of the energy cascade theory, and outer scale effect is implemented by the von Kármán refractive power spectral density. In this study, we analyze a plane wave propagating through varying atmospheric horizontal paths. An important consideration for the laser beam propagation of long distances is the random variations in the refractive index due to atmospheric turbulence. To characterize the random behavior, statistical analysis of the phase data and related metrics are examined at the output signal. We train three different machine learning algorithms in tensorflow library with the data at varying propagation lengths, outer scale lengths, and levels of turbulence intensity to predict statistical parameters that describe the atmospheric turbulence effects on laser propagation. tensorflow is an interface for demonstrating machine learning algorithms and an implementation for executing such algorithms on a wide variety of heterogeneous systems, ranging from mobile devices such as phones and tablets to large-scale distributed systems and thousands of computational devices such as GPU cards. The library contains a wide variety of algorithms including training and inference algorithms for deep neural network models. Therefore, it has been used for deploying machine learning systems in many fields including speech recognition, computer vision, natural language processing, and text mining.

A Survey on Bayesian Nonparametric Learning

Bayesian (machine) learning has been playing a significant role in machine learning for a long time due to its particular ability to embrace uncertainty, encode prior knowledge, and endow interpretability. On the back of Bayesian learning’s great success, Bayesian nonparametric learning (BNL) has emerged as a force for further advances in this field due to its greater modelling flexibility and representation power. Instead of playing with the fixed-dimensional probabilistic distributions of Bayesian learning, BNL creates a new “game” with infinite-dimensional stochastic processes. BNL has long been recognised as a research subject in statistics, and, to date, several state-of-the-art pilot studies have demonstrated that BNL has a great deal of potential to solve real-world machine-learning tasks. However, despite these promising results, BNL has not created a huge wave in the machine-learning community. Esotericism may account for this. The books and surveys on BNL written by statisticians are overcomplicated and filled with tedious theories and proofs. Each is certainly meaningful but may scare away new researchers, especially those with computer science backgrounds. Hence, the aim of this article is to provide a plain-spoken, yet comprehensive, theoretical survey of BNL in terms that researchers in the machine-learning community can understand. It is hoped this survey will serve as a starting point for understanding and exploiting the benefits of BNL in our current scholarly endeavours. To achieve this goal, we have collated the extant studies in this field and aligned them with the steps of a standard BNL procedure—from selecting the appropriate stochastic processes through manipulation to executing the model inference algorithms. At each step, past efforts have been thoroughly summarised and discussed. In addition, we have reviewed the common methods for implementing BNL in various machine-learning tasks along with its diverse applications in the real world as examples to motivate future studies.

Simultaneous clustering and feature selection via nonparametric Pitman–Yor process mixture models

Mixture models constitute one of the most important machine learning approaches. Indeed, they can be considered as the workhorse of generative machine learning. The majority of existing works consider mixtures of Gaussians. Unlike these works, this paper concentrates on nonparametric Bayesian models with Dirichlet-based mixtures. In particular, we consider the case when a Pitman–Yor process prior is adopted. Two central problems when considering such mixtures can be regarded as selecting ‘meaningful’ (or relevant) features and estimating the model’s parameters. We develop an efficient algorithm for model inference, based on the collapsed variational Bayes framework with 0th-order Taylor approximation. The merits and efficacy of the proposed nonparametric Bayesian model are demonstrated via challenging applications that concern real-world data clustering and 3D objects recognition.

Graphical Model Market Maker for Combinatorial Prediction Markets


 
 
 We describe algorithms for use by prediction markets in forming a crowd consensus joint probability distribution over thousands of related events. Equivalently, we describe market mechanisms to efficiently crowdsource both structure and parameters of a Bayesian network. Prediction markets are among the most accurate methods to combine forecasts; forecasters form a consensus probability distribution by trading contingent securities. A combinatorial prediction market forms a consensus joint distribution over many related events by allowing conditional trades or trades on Boolean combinations of events. Explicitly representing the joint distribution is infeasible, but standard inference algorithms for graphical probability models render it tractable for large numbers of base events. We show how to adapt these algorithms to compute expected assets conditional on a prospective trade, and to find the conditional state where a trader has minimum assets, allowing full asset reuse. We compare the performance of three algorithms: the straightforward algorithm from the DAGGRE (Decomposition-Based Aggregation) prediction market for geopolitical events, the simple block-merge model from the SciCast market for science and technology forecasting, and a more sophisticated algorithm we developed for future markets.
 
 

ST-SAGE

With the rapid development of location-based social networks (LBSNs), spatial item recommendation has become an important mobile application, especially when users travel away from home. However, this type of recommendation is very challenging compared to traditional recommender systems. A user may visit only a limited number of spatial items, leading to a very sparse user-item matrix. This matrix becomes even sparser when the user travels to a distant place, as most of the items visited by a user are usually located within a short distance from the user’s home. Moreover, user interests and behavior patterns may vary dramatically across different time and geographical regions. In light of this, we propose ST-SAGE, a spatial-temporal sparse additive generative model for spatial item recommendation in this article. ST-SAGE considers both personal interests of the users and the preferences of the crowd in the target region at the given time by exploiting both the co-occurrence patterns and content of spatial items. To further alleviate the data-sparsity issue, ST-SAGE exploits the geographical correlation by smoothing the crowd’s preferences over a well-designed spatial index structure called the spatial pyramid . To speed up the training process of ST-SAGE, we implement a parallel version of the model inference algorithm on the GraphLab framework. We conduct extensive experiments; the experimental results clearly demonstrate that ST-SAGE outperforms the state-of-the-art recommender systems in terms of recommendation effectiveness, model training efficiency, and online recommendation efficiency.

Probabilistic abductive logic programming using Dirichlet priors

Probabilistic programming is an area of research that aims to develop general inference algorithms for probabilistic models expressed as probabilistic programs whose execution corresponds to inferring the parameters of those models. In this paper, we introduce a probabilistic programming language (PPL) based on abductive logic programming for performing inference in probabilistic models involving categorical distributions with Dirichlet priors. We encode these models as abductive logic programs enriched with probabilistic definitions and queries, and show how to execute and compile them to boolean formulas. Using the latter, we perform generalized inference using one of two proposed Markov Chain Monte Carlo (MCMC) sampling algorithms: an adaptation of uncollapsed Gibbs sampling from related work and a novel collapsed Gibbs sampling (CGS). We show that CGS converges faster than the uncollapsed version on a latent Dirichlet allocation (LDA) task using synthetic data. On similar data, we compare our PPL with LDA-specific algorithms and other PPLs. We find that all methods, except one, perform similarly and that the more expressive the PPL, the slower it is. We illustrate applications of our PPL on real data in two variants of LDA models (Seed and Cluster LDA), and in the repeated insertion model (RIM). In the latter, our PPL yields similar conclusions to inference with EM for Mallows models.