Mean-field Inference Research Articles

Human Interaction Understanding with Consistency-Aware Learning.

Compared with the progress made on human activity classification, much less success has been achieved on human interaction understanding (HIU). Apart from the latter task is much more challenging, the main causation is that recent approaches learn human interactive relations via shallow graphical representations, which are inadequate to model complicated human interactive-relations. This paper proposes a deep consistency-aware framework aiming at tackling the grouping and labelling inconsistencies in HIU. This framework consists of three components, including a backbone CNN to extract image features, a factor graph network to implicitly learn higher-order consistencies among labelling and grouping variables, and a consistency-aware reasoning module to explicitly enforcing consistencies. The last module is inspired by our key observation that the consistency-aware reasoning bias can be embedded into an energy function or a particular loss function, minimizing which delivers consistent predictions. An efficient mean-field inference algorithm is proposed, such that all modules of our network could be trained in an end-to-end fashion. Experimental results demonstrate that the two proposed consistency-learning modules complement each other, and both make considerable contributions in achieving leading performance on three benchmarks of HIU. The effectiveness of the proposed approach is further validated by experiments on detecting human-object interactions.

Non-additive image steganographic framework based on variational inference in Markov Random Fields

The majority of image steganography methods lie within the additive scheme that assumes the pixel modifications are independent. Whereas, adapting the embedding modifications based on the mutual embedding impact of neighboring pixels reduce changes in the higher-order statistics of the cover imposed by embedding. However, non-additive schemes are more challenging due to the lack of practical embedding codes that minimize an arbitrary distortion function. In this paper, we propose a general non-additive image steganographic framework based on Markov Random Fields (MRF) which generalizes a few existing research in many aspects and offers an elegant way to model the mutual dependencies between neighborhood modifications in terms of pairwise cliques. The proposed model satisfies spatial coherence by penalizing the desynchronized embedding changes, which orients the changes towards the direction of the majority of changes in the neighbors. Mean Field (MF) inference is used to iteratively estimate the marginal probability at each pixel based on its neighbors so that the original interactions are information projected to the resulting marginals. MF replaces the values of variables with expectations so the result is compatible with practical embedding methods.The proposed framework can be applied to any additive scheme; the initial cost assignment is done by an additive method and MRF modeling defines a non-additive distortion related to statistical detectability that encourages the adjacent changes to synchronize. We study our framework in both symmetric and asymmetric schemes. In the symmetric scheme, a parallel MF is used as an adaptive filter to encourage synchronized embedding changes. The framework can upgrade to an asymmetric scheme that reduces the detectability by conditioning embedding based on the message using clamping exerted in a structural MF. Therefore, our framework is compatible with and generalizes existing non-additive methods, and as shown in experimental results, outperforms the state-of-the-art in both schemes.

A Deep Extractor for Visual Rail Surface Inspection

Rail surface inspection using a visual inspection system is an important task in the maintenance of railway networks. Recent inspection approaches have attempted to harness low-level features to identify visual defects. These methods' main issues are the limitations of low-level features and prior information about rail surfaces that show a wide variety of visual appearances on dynamic backgrounds. To overcome these problems, we propose a deep extractor (DE) that combines the strengths of fully convolutional networks and conditional random fields (CRFs), so that the deep network can learn the abstract features needed for the specified inspection task. Specifically, motivated by fully convolutional networks, a bilateral fully convolutional network containing two branches is proposed here: one encode-decode branch focuses semantic meaning; the other decode-encode branch encodes information on tiny defects. The two branches are aggregated to obtain a high-level feature map. Furthermore, the mean-field inference of a dense CRF with Gaussian pairwise potentials formulated as a recurrent neural network (CRF-RNN) is introduced to contribute to smoothing constraints and obtaining a fine-grained inspection result. By doing so, it can avoid offline post-processing and further lead the whole deep architecture to achieve end-to-end deep learning. Compared with classical approaches, our approach outperforms state-of-art approaches in analyzing two publicly available datasets.

Relational Reasoning for Group Activity Recognition via Self-Attention Augmented Conditional Random Field.

This paper presents a new relational network for group activity recognition. The essence of the network is to integrate conditional random fields (CRFs) with self-attention to infer the temporal dependencies and spatial relationships of the actors. This combination can take advantage of the capability of CRFs in modelling the actors' features that depend on each other and the capability of self-attention in learning the temporal evolution and spatial relational contexts of every actor in videos. Additionally, there are two distinct facets of our CRF and self-attention. First, the pairwise energy of the new CRF relies on both of the temporal self-attention and spatial self-attention, which apply the self-attention mechanism to the features in time and space, respectively. Second, to address both local and non-local relationships in group activities, the spatial self-attention takes account of a collection of cliques with different scales of spatial locality. The associated mean-field inference thereafter can thus be reformulated as a self-attention network to generate the relational contexts of the actors and their individual action labels. Lastly, a bidirectional universal transformer encoder (UTE) is utilized to aggregate the forward and backward temporal context information, scene information and relational contexts for group activity recognition. A new loss function is also employed, consisting of not only the cost for the classification of individual actions and group activities, but also a contrastive loss to address the miscellaneous relational contexts between actors. Simulations show that the new approach can surpass previous works on four commonly used datasets.

On the Robustness of Semantic Segmentation Models to Adversarial Attacks.

Deep Neural Networks (DNNs) have demonstrated exceptional performance on most recognition tasks such as image classification and segmentation. However, they have also been shown to be vulnerable to adversarial examples. This phenomenon has recently attracted a lot of attention but it has not been extensively studied on multiple, large-scale datasets and structured prediction tasks such as semantic segmentation which often require more specialised networks with additional components such as CRFs, dilated convolutions, skip-connections and multiscale processing. In this paper, we present what to our knowledge is the first rigorous evaluation of adversarial attacks on modern semantic segmentation models, using two large-scale datasets. We analyse the effect of different network architectures, model capacity and multiscale processing, and show that many observations made on the task of classification do not always transfer to this more complex task. Furthermore, we show how mean-field inference in deep structured models, multiscale processing (and more generally, input transformations) naturally implement recently proposed adversarial defenses. Our observations will aid future efforts in understanding and defending against adversarial examples. Moreover, in the shorter term, we show how to effectively benchmark robustness and show which segmentation models should currently be preferred in safety-critical applications due to their inherent robustness.

Context from within: Hierarchical context modeling for semantic segmentation

Conditional Random Fields (CRFs) have been widely adopted in conjunction with Fully Convolutional Networks (FCNs) to model and integrate contextual information in the semantic segmentation procedure. In contrast to existing approaches applying CRFs in parallel or in cascade with FCNs, we propose a new paradigm to incorporate CRFs deeper inside the architecture of FCNs to model the context exhibited within the middle layers of an FCN. We approximate the mean-field inference process of a dense CRF as a multi-dimensional Gated Recurrent Unit (GRU) layer, termed CRF-GRU layer, effectively extracting intermediate context within an FCN. More importantly, multiple CRF-GRU layers can be injected into an FCN to model hierarchical contexts presented in multiple middle layers, showing competitive results on the PASCAL VOC 2012 and PASCAL-Context datasets. Secondly, we contribute a new approach to automatically learn, from the training data, the optimal segmentation architecture of the FCN with multiple CRF-GRU layers injected. The proposed approach relies on Genetic Evolution Strategies to allow the existing architecture to iteratively evolve towards higher accuracy instances. The discovered network not only outperforms state-of-the-art segmentation techniques, but also provides exciting new insights into the design of the segmentation networks.

Probabilistic inference of binary Markov random fields in spiking neural networks through mean-field approximation

Recent studies have suggested that the cognitive process of the human brain is realized as probabilistic inference and can be further modeled by probabilistic graphical models like Markov random fields. Nevertheless, it remains unclear how probabilistic inference can be implemented by a network of spiking neurons in the brain. Previous studies have tried to relate the inference equation of binary Markov random fields to the dynamic equation of spiking neural networks through belief propagation algorithm and reparameterization, but they are valid only for Markov random fields with limited network structure. In this paper, we propose a spiking neural network model that can implement inference of arbitrary binary Markov random fields. Specifically, we design a spiking recurrent neural network and prove that its neuronal dynamics are mathematically equivalent to the inference process of Markov random fields by adopting mean-field theory. Furthermore, our mean-field approach unifies previous works. Theoretical analysis and experimental results, together with the application to image denoising, demonstrate that our proposed spiking neural network can get comparable results to that of mean-field inference.

Gaussian Mean Field Regularizes by Limiting Learned Information.

Variational inference with a factorized Gaussian posterior estimate is a widely-used approach for learning parameters and hidden variables. Empirically, a regularizing effect can be observed that is poorly understood. In this work, we show how mean field inference improves generalization by limiting mutual information between learned parameters and the data through noise. We quantify a maximum capacity when the posterior variance is either fixed or learned and connect it to generalization error, even when the KL-divergence in the objective is scaled by a constant. Our experiments suggest that bounding information between parameters and data effectively regularizes neural networks on both supervised and unsupervised tasks.

Water Body Extraction From Very High-Resolution Remote Sensing Imagery Using Deep U-Net and a Superpixel-Based Conditional Random Field Model

Water body extraction (WBE) has attracted considerable attention in the field of remote sensing image analysis. Herein, we present an enhanced deep convolutional encoder–decoder (DCED) network (or Deep U-Net) specifically tailored to WBE from remote sensing images by applying superpixel segmentation and conditional random fields (CRFs). First, we preclassify the entire remote sensing image into the water and nonwater areas via Deep U-Net, using the results of class membership probabilities as the unary potential in the CRF model. The pairwise potential of CRF is defined by a linear combination of Gaussian kernels, which forms a fully connected neighbor structure. Next, regional restriction is incorporated into the approach to enhance the consistency of the connected area. We use the simple linear iterative clustering algorithm to generate superpixels and correct the binary classification results by calculating their average posterior probabilities. Finally, a highly efficient approximate inference algorithm, mean-field inference, is generated for the final model. The results from the experimental application to GaoFen-2 images and WorldView-2 images demonstrate that the proposed approach exhibits competitive quantitative and qualitative performance, which effectively reduces salt-and-pepper noise and retains the edge structures of water bodies. Compared to existing state-of-the-art methods, our proposed method achieves superior final results.

Object codetection based on a higher-order conditional random field

In recent years, codetecting objects through the use of contextual information across multiple images has attracted considerable attention. We introduce an object codetection method that exploits contextual information among multiple images through a higher-order conditional random field (CRF). First, we obtain object candidates from each image of a test set by using a pretrained detector. Second, we feed the object candidates into a higher-order CRF that captures the appearance similarity using pairwise potentials and object category co-occurrence constraints using higher-order potentials. Finally, we jointly predict the category labels of all object candidates through the mean field inference in the CRF. Experimental results on the Caltech Pedestrian, PASCAL VOC 2007, PASCAL VOC 2012, and COCO datasets demonstrate the effectiveness of the proposed method compared to the baseline method.

Efficient Graph Cut Optimization for Full CRFs with Quantized Edges.

Fully connected pairwise Conditional Random Fields (Full-CRF) with Gaussian edge weights can achieve superior results compared to sparsely connected CRFs. However, traditional methods for Full-CRFs are too expensive. Previous work develops efficient approximate optimization based on mean field inference, which is a local optimization method and can be far from the optimum. We propose efficient and effective optimization based on graph cuts for Full-CRFs with quantized edge weights. To quantize edge weights, we partition the image into superpixels and assume that the weight of an edge between any two pixels depends only on the superpixels these pixels belong to. Our quantized edge CRF is an approximation to the Gaussian edge CRF, and gets closer to it as superpixel size decreases. Being an approximation, our model offers an intuition about the regularization properties of the Guassian edge Full-CRF. For efficient inference, we first consider the two-label case and develop an approximate method based on transforming the original problem into a smaller domain. Then we handle multi-label CRF by showing how to implement expansion moves. In both binary and multi-label cases, our solutions have significantly lower energy compared to that of mean field inference. We also show the effectiveness of our approach on semantic segmentation task.

Adaptive appearance separation for interactive image segmentation based on Dense CRF

Interactive segmentation has recently become a hot topic for its wide application. The authors propose an efficacious appearance separation model for interactive binary segmentation, which incorporates the difference of foreground and background colour models and the difference of corresponding geodesic models into the popular densely connected conditional random field (Dense CRF) framework. The proposed method can adaptively set relevant parameter values in this framework according to the characteristics of target images in a per-image manner, therefore, it gets rid of the dependence on specific datasets. After accomplishing a mean-field inference, the authors are able to get satisfactory results without the time-consuming parameter learning process and multiple iterative optimisations. Overall, the proposed approach is highly efficient and mitigates the contradiction between accuracy and segmentation efficiency. In addition, the proposed approach reduces the efforts of scribble-style interaction from users. The experimental results on three famous datasets show that the proposed method is superior to the other five new algorithms released in recent years regarding accuracy, and is faster than or close to them in runtime.

Exploiting context based on CNN and coding representations for pedestrian co-detection

The exploitation of contextual information among multiple images has been proven significant to improve detection performance by object co-detection methods. In this paper, we propose a pedestrian co-detection method that combines the strengths of convolutional neural networks (CNNs) and locality-constrained linear coding (LLC) in a unified conditional random field (CRF) model. First, we obtain object candidates by using a region proposal network (RPN) in Faster R-CNN. Second, we build a fully connected CRF that consists of unary potentials on individual object candidates and two types of pairwise potentials on pairs of object candidates. The unary potential is computed independently for each object candidate by using the baseline method. The pairwise potentials consist of multiscale CNN and LLC representation-based potentials, which contribute to the capturing of relationships among object candidates in all the test images. Finally, we jointly predict the category labels of all the object candidates through the mean field inference in the CRF. We evaluated the proposed method on the ETH, Caltech, and INRIA Pedestrian datasets. The experimental results demonstrate the effectiveness of the proposed method as compared to the baseline method.

Semantic scene completion with dense CRF from a single depth image

Abstract Scene understanding is a significant research topic in computer vision, especially for robots to understand their environment intelligently. Semantic scene segmentation can help robots to identify the objects that are present in their surroundings, while semantic scene completion can enhance the ability of the robot to infer the object shape, which is pivotal for several high-level tasks. With dense Conditional Random Field (CRF), one key issue is how to construct the long-range interactions between nodes with Gaussian pairwise potentials. Another issue is what effective and efficient inference algorithms can be adapted to resolve the optimization. In this paper, we focus on semantic scene segmentation and completion optimization technology simultaneously using dense CRF based on a single depth image only. Firstly, we convert the single depth image into different down-sampled Truncated Signed Distance Function (TSDF) or flipped TSDF voxel formats, and formulate the pairwise potentials terms with such a representation. Secondly, we use the output results of an end-to-end 3D convolutional neural network named SSCNet to obtain the unary potentials. Finally, we pursue the efficiency of different CRF inference algorithms (the mean-field inference, the negative semi-definite specific difference of convex relaxation, the proximal minimization of linear programming and its variants, etc.). The proposed dense CRF and inference algorithms are evaluated on three different datasets (SUNCG, NYU, and NYUCAD). Experimental results demonstrate that the voxel-level intersection over union (IoU) of predicted voxel’s semantic and completion can reach to state-of-the-art. Specifically, for voxel semantic segmentation, the highest IoU improvements are 2.6%, 1.3%, 3.1%, and for scene completion, the highest IoU improvements are 2.5%, 3.7%, 5.4%, respectively for SUNCG, NYU, and NYUCAD datasets.

Latent subspace modeling of sequential data under the maximum entropy discrimination framework

Abstract Hidden Markov models (HMMs) are a popular approach for modeling continuous sequential data, typically based on the assumption of Gaussian-distributed observations. A significant issue HMMs with Gaussian conditional densities are confronted with concerns effectively modeling high-dimensional observations, without getting prone to overfitting or singularities. To this end, one can resort to extracting lower-dimensional latent variable representations of the observed high-dimensional data, as part of the inference algorithm of the postulated HMM. Factor analysis (FA) is a well-established linear latent variable scheme that can be employed for this purpose; its functionality consists in modeling the covariances between the elements of multivariate observations under a set of linear assumptions. Recently, it has been proposed that FA can be effectively generalized under an efficient large-margin Bayesian inference perspective, namely maximum entropy discrimination (MED). This work capitalizes on these recent findings to derive an effective HMM-driven sequential data modeling framework for high-dimensional data. Our proposed approach extracts lower-dimensional latent variable representations of observed high-dimensional data, taking into account the large-margin principle. On this basis, it postulates that the data temporal dynamics are conditional to the inferred values of these latent variables. We devise efficient mean-field inference algorithms for our model, and exhibit its advantages through a set of experiments.

Combined Message Passing Based SBL With Dirichlet Process Prior for Sparse Signal Recovery With Multiple Measurement Vectors

This paper concerns the problem of sparse signal recovery with multiple measurement vectors, where the sparse signal vectors share multiple supports (i.e., the signal vectors can be clustered and the vectors in a cluster share a common support) and the prior knowledge on the supports of the vectors is unknown. This problem can be solved using sparse Bayesian learning (SBL) with Dirichlet process (DP) as hyper-prior, which is named DP-SBL in this paper. This paper aims to design efficient inference algorithms. The variational inference for DP mixtures, in particular mean field (MF) inference, has been studied, and applying it to the problem in this paper leads to an MF-DP-SBL algorithm. In this paper, we propose a combined message passing (CMP) approach, where a factor graph representation is designed to enable a more efficient implementation with both the MF and approximate message passing (AMP), leading to a CMP-DP-SBL algorithm. It is shown that, compared with MF-DP-SBL and CMP-DP-SBL delivers the same or even better performance with significantly lower complexity. As an example, we apply it to massive MIMO channel estimation where, due to the large number of antennas deployed at the base station, the channel impulse responses measured at receive antennas can share multiple supports. It is shown that CMP-DP-SBL delivers considerably better performance than existing algorithms.

Improving Semantic Image Segmentation With a Probabilistic Superpixel-Based Dense Conditional Random Field

Deep convolutional neural networks (DCNNs) have been driving significant advances in semantic image segmentation due to their powerful feature representation for recognition. However, their performance in preserving object boundaries is still not satisfactory. Visual mechanism theory indicates that image segmentation tasks require not only recognition, like DCNNs, but also local visual attention capability. Considering that superpixel is good at grasping detailed local structure, we propose a probabilistic superpixel-based dense conditional random field model (PSP-CRF) to refine label assignments as a post-processing optimization method. First, the well-known fully convolutional networks (FCN) and Deeplab-ResNet are employed to produce coarse prediction probabilistic maps at each pixel. Second, we construct a fully connected CRF model based on the PSP generated by the simple linear iterative clustering algorithm. In our approach, an effective refining algorithm with entropy is developed to convert the pixel-level appearance and position features to the normalized PSP, which works well for CRF. Third, our method optimizes the PSP-CRF to obtain the final label assignment results by employing a highly efficient mean field inference algorithm and some quadratic programming relaxation related algorithms. The experiments on the PASCAL VOC segmentation dataset demonstrate the effectiveness of our methods which can improve the segmentation performance of DCNNs to 82% in mIoU while increasing the computational efficiency by 47%.

BUILDING DETECTION USING AERIAL IMAGES AND DIGITAL SURFACE MODELS

Abstract. In this paper a method for building detection in aerial images based on variational inference of logistic regression is proposed. It consists of three steps. In order to characterize the appearances of buildings in aerial images, an effective bag-of-Words (BoW) method is applied for feature extraction in the first step. In the second step, a classifier of logistic regression is learned using these local features. The logistic regression can be trained using different methods. In this paper we adopt a fully Bayesian treatment for learning the classifier, which has a number of obvious advantages over other learning methods. Due to the presence of hyper prior in the probabilistic model of logistic regression, approximate inference methods have to be applied for prediction. In order to speed up the inference, a variational inference method based on mean field instead of stochastic approximation such as Markov Chain Monte Carlo is applied. After the prediction, a probabilistic map is obtained. In the third step, a fully connected conditional random field model is formulated and the probabilistic map is used as the data term in the model. A mean field inference is utilized in order to obtain a binary building mask. A benchmark data set consisting of aerial images and digital surfaced model (DSM) released by ISPRS for 2D semantic labeling is used for performance evaluation. The results demonstrate the effectiveness of the proposed method.

Resummed mean-field inference for strongly coupled data.

We present a resummed mean-field approximation for inferring the parameters of an Ising or a Potts model from empirical, noisy, one- and two-point correlation functions. Based on a resummation of a class of diagrams of the small correlation expansion of the log-likelihood, the method outperforms standard mean-field inference methods, even when they are regularized. The inference is stable with respect to sampling noise, contrarily to previous works based either on the small correlation expansion, on the Bethe free energy, or on the mean-field and Gaussian models. Because it is mostly analytic, its complexity is still very low, requiring an iterative algorithm to solve for N auxiliary variables, that resorts only to matrix inversions and multiplications. We test our algorithm on the Sherrington-Kirkpatrick model submitted to a random external field and large random couplings, and demonstrate that even without regularization, the inference is stable across the whole phase diagram. In addition, the calculation leads to a consistent estimation of the entropy of the data and allows us to sample form the inferred distribution to obtain artificial data that are consistent with the empirical distribution.

Maximum entropy discrimination factor analyzers

Devising generative models that allow for inferring low dimensional latent feature representations of high-dimensional observations is a significant problem in statistical machine learning. Factor analysis (FA) is a well-established linear latent variable scheme addressing this problem by modeling the covariances between the elements of multivariate observations under a set of linear assumptions. FA is closely related to principal components analysis (PCA), and might be considered as a generalization of both PCA and its probabilistic version, PPCA. Recently, the invention of Gaussian process latent variable models (GP-LVMs) has given rise to a whole new family of latent variable modeling schemes that generalize FA under a nonparametric Bayesian inference framework. In this work, we examine generalization of FA models under a different Bayesian inference perspective. Specifically, we propose a large-margin formulation of FA under the maximum entropy discrimination (MED) framework. The MED framework integrates the large-margin principle with Bayesian posterior inference in an elegant and computationally efficient fashion, allowing to leverage existing high-performance solvers for convex optimization problems. We devise efficient mean-field inference algorithms for our model, and exhibit its advantages by evaluating it in a number of diverse application scenarios, dealing with high-dimensional data classification and reconstruction.