Conditional Random Fields Inference Research Articles

Improved Node and Arc Multiplicity Estimation in De Bruijn Graphs Using Approximate Inference in Conditional Random Fields.

In de novo genome assembly using short Illumina reads, the accurate determination of node and arc multiplicities in a de Bruijn graph has a large impact on the quality and contiguity of the assembly. The multiplicity estimates of nodes and arcs guide the cleaning of the de Bruijn graph by identifying spurious nodes and arcs that correspond to sequencing errors. Additionally, they can be used to guide repeat resolution. Here, we model the entire de Bruijn graph and the accompanying read coverage information with a single Conditional Random Field (CRF) model. We show that approximate inference using Loopy Belief Propagation (LBP) on our model improves multiplicity assignment accuracy within feasible runtimes. The order in which messages are passed has a large influence on the speed of LBP convergence. Little theoretical guarantees exist and the conditions for convergence are not easily checked as our CRF model contains higher-order interactions. Therefore, we also present an empirical evaluation of several message passing schemes that may guide future users of LBP on CRFs with higher-order interactions in their choice of message passing scheme.

Application of Conditional Random Field Model Based on Machine Learning in Online and Offline Integrated Educational Resource Recommendation

It is of great significance to mine the learning resources that learners are interested in from massive data and recommend appropriate educational resources to them according to the characteristics of students. To improve the accuracy of educational resource recommendation, this paper proposes an educational resource recommendation system based on a graph attention network and conditional random field fusion model. It builds all comments for each student and educational resource into a comment graph. Through the graph's topological structure to capture the network and the dependency between words in the commentary text, the adjacency information of each node is aggregated by the graph attention network based on connection relation. After the graph attention network layer, the conditional random field inference layer is added. The label sequence with the highest probability is output by the dependent random field inference layer, which is taken as the final recommendation result of the model. Experimental results show that the proposed algorithm has better performance in accuracy and diversity than the traditional recommendation algorithm.

End-to-End Learning Deep CRF Models for Multi-Object Tracking Deep CRF Models

By bundling multiple complex sub-problems into a unified framework, end-to-end deep learning frameworks reduce the need for hand engineering or tuning of parameters for each component, and optimize different modules jointly to ensure the generalization of the whole deep architecture. Despite tremendous success in numerous computer vision tasks, end-to-end learnings for multi-object tracking (MOT), especially for the assignment problem in data association, have been surprisingly less investigated mainly due to the lack of available training data. Furthermore, it is challenging to discriminate target objects under mutual occlusions or to reduce identity switches in crowded scenes. To tackle these challenges, this paper proposes learning deep conditional random field (CRF) networks, aiming to model the assignment costs as unary potentials and the long-term dependencies among detection results as pairwise potentials. Specifically, we use a bidirectional long short-term memory (LSTM) network to encode the long-term dependencies. We pose the CRF inference as a recurrent neural network learning process using the standard gradient descent algorithm, where unary and pairwise potentials are jointly optimized in an end-to-end manner. Extensive experiments are conducted on the challenging MOT datasets including MOT15, MOT16 and MOT17, and the results show that the proposed algorithm performs favorably against the state-of-the-art methods.

A unified convolutional neural network integrated with conditional random field for pipe defect segmentation

AbstractSemantic segmentation of closed‐circuit television (CCTV) images can facilitate automatic severity assessment of sewer pipe defects by assigning defect labels to each pixel on the image, from which defect types, locations, and geometric information can be obtained. In this study, a unified neural network, namely DilaSeg‐CRF, is proposed by fully integrating a deep convolutional neural network (CNN) with dense conditional random field (CRF) for improving the segmentation accuracy. First, DilaSeg is constructed with dilated convolution and multiscale techniques for producing feature maps with high resolution. The steps of the dense CRF inference algorithm are converted into CNN operations, which are then formulated as recurrent neural network (RNN) layers. The DilaSeg‐CRF is proposed by integrating DilaSeg with the RNN layers. Images containing three common types of sewer defects are collected from CCTV inspection videos and are annotated with ground truth labels, after which the proposed models are trained and evaluated. Experiments demonstrate that the end‐to‐end trainable DilaSeg‐CRF can improve the segmentation significantly, with an increase of 32% and 20% in mean intersection over union (mIoU) values compared with fully convolutional network (FCN‐8s) and DilaSeg, respectively. Our proposed DilaSeg‐CRF also achieves faster inference speed than FCN and eliminates the manual postprocessing for refining the segmentation results.

Semantic scene completion with dense CRF from a single depth image

Abstract Scene understanding is a significant research topic in computer vision, especially for robots to understand their environment intelligently. Semantic scene segmentation can help robots to identify the objects that are present in their surroundings, while semantic scene completion can enhance the ability of the robot to infer the object shape, which is pivotal for several high-level tasks. With dense Conditional Random Field (CRF), one key issue is how to construct the long-range interactions between nodes with Gaussian pairwise potentials. Another issue is what effective and efficient inference algorithms can be adapted to resolve the optimization. In this paper, we focus on semantic scene segmentation and completion optimization technology simultaneously using dense CRF based on a single depth image only. Firstly, we convert the single depth image into different down-sampled Truncated Signed Distance Function (TSDF) or flipped TSDF voxel formats, and formulate the pairwise potentials terms with such a representation. Secondly, we use the output results of an end-to-end 3D convolutional neural network named SSCNet to obtain the unary potentials. Finally, we pursue the efficiency of different CRF inference algorithms (the mean-field inference, the negative semi-definite specific difference of convex relaxation, the proximal minimization of linear programming and its variants, etc.). The proposed dense CRF and inference algorithms are evaluated on three different datasets (SUNCG, NYU, and NYUCAD). Experimental results demonstrate that the voxel-level intersection over union (IoU) of predicted voxel’s semantic and completion can reach to state-of-the-art. Specifically, for voxel semantic segmentation, the highest IoU improvements are 2.6%, 1.3%, 3.1%, and for scene completion, the highest IoU improvements are 2.5%, 3.7%, 5.4%, respectively for SUNCG, NYU, and NYUCAD datasets.

ST-IRGS: A Region-Based Self-Training Algorithm Applied to Hyperspectral Image Classification and Segmentation

The problem of limited labeled training samples is challenging for the classification of remote sensing imagery. We develop a joint classification and segmentation algorithm to address this problem. Our algorithm combines semisupervised learning and conditional random fields (CRFs) into a single framework. The multimodal Gaussian maximum-likelihood classifier is used to estimate the probabilities for the unary potentials of the CRF. Unlike traditional methods based on random fields, region merging is concatenated with the CRF inference to reduce the number of nodes iteratively. Moreover, a semisupervised technique called self-training is used, which iteratively enlarges the training sample set and retrains the classifier. The selection of training samples is based on the region information, so that the risk of assigning wrong labels is largely reduced. The proposed algorithm is applied to hyperspectral image classification, and results on benchmark data sets show that the proposed algorithm significantly improves classification performance after using self-training, and outperforms state-of-the-art spectral–spatial methods for limited labeled training samples.

A higher order conditional random field model for simultaneous classification of land cover and land use

We propose a new approach for the simultaneous classification of land cover and land use considering spatial as well as semantic context. We apply a Conditional Random Fields (CRF) consisting of a land cover and a land use layer. In the land cover layer of the CRF, the nodes represent super-pixels; in the land use layer, the nodes correspond to objects from a geospatial database. Intra-layer edges of the CRF model spatial dependencies between neighbouring image sites. All spatially overlapping sites in both layers are connected by inter-layer edges, which leads to higher order cliques modelling the semantic relation between all land cover and land use sites in the clique. A generic formulation of the higher order potential is proposed. In order to enable efficient inference in the two-layer higher order CRF, we propose an iterative inference procedure in which the two classification tasks mutually influence each other. We integrate contextual relations between land cover and land use in the classification process by using contextual features describing the complex dependencies of all nodes in a higher order clique. These features are incorporated in a discriminative classifier, which approximates the higher order potentials during the inference procedure. The approach is designed for input data based on aerial images. Experiments are carried out on two test sites to evaluate the performance of the proposed method. The experiments show that the classification results are improved compared to the results of a non-contextual classifier. For land cover classification, the result is much more homogeneous and the delineation of land cover segments is improved. For the land use classification, an improvement is mainly achieved for land use objects showing non-typical characteristics or similarities to other land use classes. Furthermore, we have shown that the size of the super-pixels has an influence on the level of detail of the classification result, but also on the degree of smoothing induced by the segmentation method, which is especially beneficial for land cover classes covering large, homogeneous areas.

Exploring Context with Deep Structured Models for Semantic Segmentation.

We propose an approach for exploiting contextual information in semantic image segmentation, and particularly investigate the use of patch-patch context and patch-background context in deep CNNs. We formulate deep structured models by combining CNNs and Conditional Random Fields (CRFs) for learning the patch-patch context between image regions. Specifically, we formulate CNN-based pairwise potential functions to capture semantic correlations between neighboring patches. Efficient piecewise training of the proposed deep structured model is then applied in order to avoid repeated expensive CRF inference during the course of back propagation. For capturing the patch-background context, we show that a network design with traditional multi-scale image inputs and sliding pyramid pooling is very effective for improving performance. We perform comprehensive evaluation of the proposed method. We achieve new state-of-the-art performance on a number of challenging semantic segmentation datasets.

CRF‐based depth refinement with hybrid depth information

A conditional random field (CRF)-based framework is proposed for depth refinement. It consists of two phases: depth impainting and depth super-resolution (SR). The CRF inference procedure uses depth information enhanced in three different ways to ensure the accurate edge of the final depth output. The experimental results show the effectiveness of the method both numerically and visually. Note that the method produces depth map with better edge than other depth impainting and SR methods do.

A Fast Method for the Segmentation of Synaptic Junctions and Mitochondria in Serial Electron Microscopic Images of the Brain.

Recent electron microscopy (EM) imaging techniques permit the automatic acquisition of a large number of serial sections from brain samples. Manual segmentation of these images is tedious, time-consuming and requires a high degree of user expertise. Therefore, there is considerable interest in developing automatic segmentation methods. However, currently available methods are computationally demanding in terms of computer time and memory usage, and to work properly many of them require image stacks to be isotropic, that is, voxels must have the same size in the X, Y and Z axes. We present a method that works with anisotropic voxels and that is computationally efficient allowing the segmentation of large image stacks. Our approach involves anisotropy-aware regularization via conditional random field inference and surface smoothing techniques to improve the segmentation and visualization. We have focused on the segmentation of mitochondria and synaptic junctions in EM stacks from the cerebral cortex, and have compared the results to those obtained by other methods. Our method is faster than other methods with similar segmentation results. Our image regularization procedure introduces high-level knowledge about the structure of labels. We have also reduced memory requirements with the introduction of energy optimization in overlapping partitions, which permits the regularization of very large image stacks. Finally, the surface smoothing step improves the appearance of three-dimensional renderings of the segmented volumes. Electronic supplementary material The online version of this article (doi:10.1007/s12021-015-9288-z) contains supplementary material, which is available to authorized users.

Sequential Labeling Using Deep-Structured Conditional Random Fields

We develop and present the deep-structured conditional random field (CRF), a multi-layer CRF model in which each higher layer's input observation sequence consists of the previous layer's observation sequence and the resulted frame-level marginal probabilities. Such a structure can closely approximate the long-range state dependency using only linear-chain or zeroth-order CRFs by constructing features on the previous layer's output (belief). Although the final layer is trained to maximize the log-likelihood of the state (label) sequence, each lower layer is optimized by maximizing the frame-level marginal probabilities. In this deep-structured CRF, both parameter estimation and state sequence inference are carried out efficiently layer-by-layer from bottom to top. We evaluate the deep-structured CRF on two natural language processing tasks: search query tagging and advertisement field segmentation. The experimental results demonstrate that the deep-structured CRF achieves word labeling accuracies that are significantly higher than the best results reported on these tasks using the same labeled training set.

Aspect coherence for graph-based semantic image labelling

In image semantic segmentation a semantic category label is associated to each image pixel. This classification problem is characterised by pixel dependencies at different scales. On a small-scale pixel correlation is related to object instance sharing, whereas on a middle- and large scale to category co-presence and relative location constraints. The contribution of this study is two-fold. First, the authors present a framework that jointly learns category appearances and pixel dependencies at different scales. Small-scale dependencies are accounted by clustering pixels into larger patches via image oversegmentation. To tackle middle-scale dependencies a conditional random field (CRF) is built over the patches. A novel strategy to exploit local patch aspect coherence is used to impose an optimised structure in the graph to have exact and efficient inference. The second contribution is a method to account for full patch neighbourhoods without introducing loops in the graphical structures. ‘Weak neighbours’ are introduced, which are patches connected in the image but not in the inference graph. They are pre-classified according to their visual appearance and their category distribution probability is then used in the CRF inference step. Experimental evidence of the validity of the method shows improvements in comparison to other works in the field.