Hierarchical Conditional Random Field Research Articles

MULTI-SOURCE MULTI-SCALE HIERARCHICAL CONDITIONAL RANDOM FIELD MODEL FOR REMOTE SENSING IMAGE CLASSIFICATION

Abstract. Fusion of remote sensing images and LiDAR data provides complimentary information for the remote sensing applications, such as object classification and recognition. In this paper, we propose a novel multi-source multi-scale hierarchical conditional random field (MSMSH-CRF) model to integrate features extracted from remote sensing images and LiDAR point cloud data for image classification. MSMSH-CRF model is then constructed to exploit the features, category compatibility of multi-scale images and the category consistency of multi-source data based on the regions. The output of the model represents the optimal results of the image classification. We have evaluated the precision and robustness of the proposed method on airborne data, which shows that the proposed method outperforms standard CRF method.

Hybrid graphical model for semantic image segmentation

To make full use of both non-causal and causal cues in natural images, we propose a hybrid hierarchical Conditional Random Field (HCRF) and Bayesian Network (BN) model for semantic image segmentation in this paper. The HCRF is used to capture non-causal relationship, such as appearance features and inter-class co-occurrence statistics, to produce initial semantic sub-scene predictions. Whereas, the BN is used to model contextual interactions for each semantic sub-scene in the form of class statistics from its neighboring regions, of which its conditional probabilities are learned automatically from training data. The learned BN structure is then used to encode the structure of contextual dependencies for sub-scenes in the initial predictions to generate final refined predictions. Experiments on the Stanford 8-class dataset and the LHI 15-class dataset show that the hybrid model outperforms pure CRF models by 2–4% in average classification accuracy.

Sign language recognition with the Kinect sensor based on conditional random fields.

Sign language is a visual language used by deaf people. One difficulty of sign language recognition is that sign instances of vary in both motion and shape in three-dimensional (3D) space. In this research, we use 3D depth information from hand motions, generated from Microsoft's Kinect sensor and apply a hierarchical conditional random field (CRF) that recognizes hand signs from the hand motions. The proposed method uses a hierarchical CRF to detect candidate segments of signs using hand motions, and then a BoostMap embedding method to verify the hand shapes of the segmented signs. Experiments demonstrated that the proposed method could recognize signs from signed sentence data at a rate of 90.4%.

Multilabel region classification and semantic linking for colon segmentation in CT colonography.

Accurate and automatic colon segmentation from CT images is a crucial step of many clinical applications in CT colonography, including computer-aided detection (CAD) of colon polyps, 3-D virtual flythrough of the colon, and prone/supine registration. However, the existence of adjacent air-filled organs such as the lung, stomach, and small intestine, and the collapse of the colon due to poor insufflation, render accurate segmentation of the colon a difficult problem. Extra-colonic components can be categorized into two types based on their 3-D connection to the colon: detached and attached extracolonic components (DEC and AEC, respectively). In this paper, we propose graph inference methods to remove extracolonic components to achieve a high quality segmentation. We first decompose each 3-D air-filled object into a set of 3-D regions. A classifier trained with region-level features can be used to identify the colon regions from noncolon regions. After removing obvious DEC, we remove the remaining DEC by modeling the global anatomic structure with an a priori topological constraint and solving a graph inference problem using semantic information provided by a multiclass classifier. Finally, we remove AEC by modeling regions within each 3-D object with a hierarchical conditional random field, solved by graph cut. Experimental results demonstrate that our method outperforms a purely discriminative learning method in detecting true colon regions, while decreasing extra-colonic components in challenging clinical data that includes collapsed cases.

Efficient Learning of Spatial Patterns with Multi-Scale Conditional Random Fields for Region-Based Classification

Automatic image classification is of major importance for a wide range of applications and is supported by a complex process that usually requires the identification of individual regions and spatial patterns (contextual information) among neighboring regions within images. Hierarchical conditional random fields (CRF) consider both multi-scale and contextual information in a unified discriminative probabilistic framework, yet they suffer from two main drawbacks. On the one hand, their current classification performance still leaves space for improvement, mostly due to the use of very simple or inappropriate pairwise energy expressions to model complex spatial patterns; on the other hand, their training remains complex, particularly for multi-class problems. In this work, we investigated alternative pairwise energy expressions to better account for class transitions and developed an efficient parameters learning strategy for the resultant expression. We propose: (i) a multi-scale CRF model with novel energies that involves information related to the multi-scale image structure; and (ii) an efficient maximum margin parameters learning procedure where the complex learning problem is decomposed into simpler individual multi-class sub-problems. During experiments conducted on several well-known satellite image data sets, the suggested multi-scale CRF exhibited between a 1% and 15% accuracy improvement compared to other works. We also found that, on different multi-scale decompositions, the total number of regions and their average size have a direct impact on the classification results.

Learning a Dictionary of Shape Epitomes with Applications to Image Labeling.

The first main contribution of this paper is a novel method for representing images based on a dictionary of shape epitomes. These shape epitomes represent the local edge structure of the image and include hidden variables to encode shift and rotations. They are learnt in an unsupervised manner from groundtruth edges. This dictionary is compact but is also able to capture the typical shapes of edges in natural images. In this paper, we illustrate the shape epitomes by applying them to the image labeling task. In other work, described in the supplementary material, we apply them to edge detection and image modeling. We apply shape epitomes to image labeling by using Conditional Random Field (CRF) Models. They are alternatives to the superpixel or pixel representations used in most CRFs. In our approach, the shape of an image patch is encoded by a shape epitome from the dictionary. Unlike the superpixel representation, our method avoids making early decisions which cannot be reversed. Our resulting hierarchical CRFs efficiently capture both local and global class co-occurrence properties. We demonstrate its quantitative and qualitative properties of our approach with image labeling experiments on two standard datasets: MSRC-21 and Stanford Background.

Robust sign language recognition by combining manual and non-manual features based on conditional random field and support vector machine

The sign language is composed of two categories of signals: manual signals such as signs and fingerspellings and non-manual ones such as body gestures and facial expressions. This paper proposes a new method for recognizing manual signals and facial expressions as non-manual signals. The proposed method involves the following three steps: First, a hierarchical conditional random field is used to detect candidate segments of manual signals. Second, the BoostMap embedding method is used to verify hand shapes of segmented signs and to recognize fingerspellings. Finally, the support vector machine is used to recognize facial expressions as non-manual signals. This final step is taken when there is some ambiguity in the previous two steps. The experimental results indicate that the proposed method can accurately recognize the sign language at an 84% rate based on utterance data.

QoE-Based Multi-Exposure Fusion in Hierarchical Multivariate Gaussian CRF

Many state-of-the-art fusion methods, combining details in images taken under different exposures into one well-exposed image, can be found in the literature. However, insufficient study has been conducted to explore how perceptual factors can provide viewers better quality of experience on fused images. We propose two perceptual quality measures: perceived local contrast and color saturation, which are embedded in our novel hierarchical multivariate Gaussian conditional random field model, to illustrate improved performance for multi-exposure fusion. We show that our method generates images with better quality than existing methods for a variety of scenes.

MULTI-SOURCE HIERARCHICAL CONDITIONAL RANDOM FIELD MODEL FOR FEATURE FUSION OF REMOTE SENSING IMAGES AND LIDAR DATA

Abstract. Feature fusion of remote sensing images and LiDAR points cloud data, which have strong complementarity, can effectively play the advantages of multi-class features to provide more reliable information support for the remote sensing applications, such as object classification and recognition. In this paper, we introduce a novel multi-source hierarchical conditional random field (MSHCRF) model to fuse features extracted from remote sensing images and LiDAR data for image classification. Firstly, typical features are selected to obtain the interest regions from multi-source data, then MSHCRF model is constructed to exploit up the features, category compatibility of images and the category consistency of multi-source data based on the regions, and the outputs of the model represents the optimal results of the image classification. Competitive results demonstrate the precision and robustness of the proposed method.

A probabilistic model with multi-dimensional features for object extraction

To identify recruitment information in different domains, we propose a novel model of hierarchical tree-structured conditional random fields (HT-CRFs). In our approach, first, the concept of a Web object (WOB) is discussed for the description of special Web information. Second, in contrast to traditional methods, the Boolean model and multi-rule are introduced to denote a one-dimensional text feature for a better representation of Web objects. Furthermore, a two-dimensional semantic texture feature is developed to discover the layout of a WOB, which can emphasize the structural attributes and the specific semantics term attributes of WOBs. Third, an optimal WOB information extraction (IE) based on HT-CRF is performed, addressing the problem of a model having an excessive dependence on the page structure and optimizing the efficiency of the model’s training. Finally, we compare the proposed model with existing decoupled approaches for WOB IE. The experimental results show that the accuracy rate of WOB IE is significantly improved and that time complexity is reduced.

Conditional random field-based image labelling combining features of pixels, segments and regions

Conditional random field (CRF)-based framework is the most popular approach to image labelling. Pixel-based CRF and segment-based CRF correspond to image representations on different scales. Hierarchical CRF models are the main technique to combine multi-scale information of an image. In this study, the authors propose a single-layered segment-based CRF, instead of multi-layered hierarchical CRF, to integrate multi-scale features of pixels, segments and regions. The unary potential associated with a segment in the CRF is determined by the features of pixels in it, instead of by the statistical features of the segment. On the other hand, the features of a pixel contain features of multiple segments it belongs to. By this means, features of pixels and segments computed at different levels are integrated naturally. Furthermore, to alleviate the problem of local minima and to capture long-range semantic context, the authors propose a region-based CRF to model co-occurrence. Compared with some existing approaches to model co-occurrence, it is relatively fast and can correct some co-occurrence constraints violation errors. Experiments on MSRC-21 database show that our model achieves comparable results to the state-of-the-art algorithms but with lower complexity.

The echo state conditional random field model for sequential data modeling

Sequential data labeling is a fundamental task in machine learning applications, with speech and natural language processing, activity recognition in video sequences, and biomedical data analysis being characteristic such examples, to name just a few. The conditional random field (CRF), a log-linear model representing the conditional distribution of the observation labels, is one of the most successful approaches for sequential data labeling and classification, and has lately received significant attention in machine learning, as it achieves superb prediction performance in a variety of scenarios. Nevertheless, existing CRF formulations do not account for temporal dependencies between the observed variables – they only postulate Markovian interdependencies between the predicted label variables. To resolve these issues, in this paper we propose a non-linear hierarchical CRF formulation that combines the power of echo state networks to extract high level temporal features with the graphical framework of CRF models, yielding a powerful and scalable probabilistic model that we apply to signal labeling tasks.

Signet: Web Information Retrieval with NE Disambiguation based on HMM and CRF

As compared to many other techniques used in natural language processing, hidden markov models (HMMs) are an extremely flexible tool and has been successfully applied to a wide variety of information extraction tasks. This work focus on webpage perceptive through model of Hierarchical Conditional Random Fields (i.e. HCRF) and offer results in free text segmentation and labeling. This paper specially addresses the problem of research community of academic people integration (SIGNET-similar interest group) through perceiving the entities of them. Keywords—HMM, HCRF, Named-Entity, SIGNET

Image Labeling and Segmentation using Hierarchical Conditional Random Field Model

The use of hierarchical Conditional Random Field model deal with the problem of labeling images . At the time of labeling a new image, selection of the nearest cluster and using the related CRF model to label this image. When one give input image, one first use the CRF model to get initial pixel labels then finding the cluster with most similar images. Then at last relabeling the input image by the CRF model associated with this cluster. This paper presents a approach to label and segment specific image having correct information.

Harmony Potentials

The Hierarchical Conditional Random Field (HCRF) model have been successfully applied to a number of image labeling problems, including image segmentation. However, existing HCRF models of image segmentation do not allow multiple classes to be assigned to a single region, which limits their ability to incorporate contextual information across multiple scales. At higher scales in the image, this representation yields an oversimplified model since multiple classes can be reasonably expected to appear within large regions. This simplified model particularly limits the impact of information at higher scales. Since class-label information at these scales is usually more reliable than at lower, noisier scales, neglecting this information is undesirable. To address these issues, we propose a new consistency potential for image labeling problems, which we call the harmony potential. It can encode any possible combination of labels, penalizing only unlikely combinations of classes. We also propose an effective sampling strategy over this expanded label set that renders tractable the underlying optimization problem. Our approach obtains state-of-the-art results on two challenging, standard benchmark datasets for semantic image segmentation: PASCAL VOC 2010, and MSRC-21.

Simultaneous spotting of signs and fingerspellings based on hierarchical conditional random fields and boostmap embeddings

A sign language consists of two types of action; signs and fingerspellings. Signs are dynamic gestures discriminated by continuous hand motions and hand configurations, while fingerspellings are a combination of continuous hand configurations. Sign language spotting is the task of detection and recognition of signs and fingerspellings in a signed utterance. The internal structures of signs and fingerspellings differ significantly. Therefore, it is difficult to spot signs and fingerspellings simultaneously. In this paper, a novel method for spotting signs and fingerspellings is proposed. It can distinguish signs, fingerspellings and non-sign patterns, and is robust to the various sizes, scales and rotations of the signer's hand. This is achieved through a hierarchical framework consisting of three steps: (1) Candidate segments of signs and fingerspellings are discriminated using a two-layer conditional random field (CRF). (2) Hand shapes of segmented signs and fingerspellings are verified using BoostMap embeddings. (3) The motions of fingerspellings are verified in order to distinguish those which have similar hand shapes and different hand motions. Experiments demonstrate that the proposed method can spot signs and fingerspellings from utterance data at rates of 83% and 78%, respectively.

Closing the Loop in Webpage Understanding

The two most important tasks in information extraction from the Web are webpage structure understanding and natural language sentences processing. However, little work has been done toward an integrated statistical model for understanding webpage structures and processing natural language sentences within the HTML elements. Our recent work on webpage understanding introduces a joint model of Hierarchical Conditional Random Fields (HCRFs) and extended Semi-Markov Conditional Random Fields (Semi-CRFs) to leverage the page structure understanding results in free text segmentation and labeling. In this top-down integration model, the decision of the HCRF model could guide the decision making of the Semi-CRF model. However, the drawback of the top-down integration strategy is also apparent, i.e., the decision of the Semi-CRF model could not be used by the HCRF model to guide its decision making. This paper proposed a novel framework called WebNLP, which enables bidirectional integration of page structure understanding and text understanding in an iterative manner. We have applied the proposed framework to local business entity extraction and Chinese person and organization name extraction. Experiments show that the WebNLP framework achieved significantly better performance than existing methods.

Extracting Places and Activities from GPS Traces Using Hierarchical Conditional Random Fields

Learning patterns of human behavior from sensor data is extremely important for high-level activity inference. This paper describes how to extract a person’s activities and significant places from traces of GPS data. The system uses hierarchically structured conditional random fields to generate a consistent model of a person’s activities and places. In contrast to existing techniques, this approach takes the high-level context into account in order to detect the significant places of a person. Experiments show significant improvements over existing techniques. Furthermore, they indicate that the proposed system is able to robustly estimate a person’s activities using a model that is trained from data collected by other persons.