Conditional Random Field Framework Research Articles

Social Listening for Product Design Requirement Analysis and Segmentation: A Graph Analysis Approach with User Comments Mining.

This study investigates customers' product design requirements through online comments from social media, and quickly translates these needs into product design specifications. First, the exponential discriminative snowball sampling method was proposed to generate a product-related subnetwork. Second, natural language processing (NLP) was utilized to mine user-generated comments, and a Graph SAmple and aggreGatE method was employed to embed the user's node neighborhood information in the network to jointly define a user's persona. Clustering was used for market and product model segmentation. Finally, a deep learning bidirectional long short-term memory with conditional random fields framework was introduced for opinion mining. A comment frequency-invert group frequency indicator was proposed to quantify all user groups' positive and negative opinions for various specifications of different product functions. A case study of smartphone design analysis is presented with data from a large Chinese online community called Baidu Tieba. Eleven layers of social relationships were snowball sampled, with 14,018 users and 30,803 comments. The proposed method produced a more reasonable user group clustering result than the conventional method. With our approach, user groups' dominating likes and dislikes for specifications could be immediately identified, and the similar and different preferences of product features by different user groups were instantly revealed. Managerial and engineering insights were also discussed.

A conditional random field framework for language process in product review mining

The Part-Of-Speech tagging is widely used in the natural language process. There are many statistical approaches in this area. The most popular one is Hidden Markov Model. In this paper, an alternative approach, linear-chain Conditional Random Fields, is introduced. The Conditional Random Fields is a factor graph approach that can naturally incorporate arbitrary, non-independent features of the input without conditional independence among the features or distributional assumptions of inputs. This paper applied the Conditional Random Fields for the car review word Part-Of-Speech tagging and then the feature extraction, which can be used as an input to an opinion mining system. To reduce the computational time, we also proposed applying the Limited-memory BFGS algorithm to train the Conditional Random Fields. Furthermore, this paper evaluated the Conditional Random Fields and the classical graph approach using the car review dataset to demonstrate that the Conditional Random Fields have a more robust result with a smaller training dataset.

Mapping large-scale and fine-grained urban functional zones from VHR images using a multi-scale semantic segmentation network and object based approach

Urban functional zones (UFZs) are essential for characterizing both urban spatial configurations and socio-economic properties and monitoring urbanization process, thus UFZs are fundamental to urban planning, management and renewal. Although many efforts in remote sensing field have been made to map UFZs, large-scale and fine-grained UFZ maps required by a broad range of urban applications are still unavailable. Existing methods generally rely on pre-determined mapping units, such as image tiles and road blocks, which significantly limit the mapping quality and the automation degree of mapping UFZs. Given that UFZs are composed of diverse geographic objects, this study proposes a novel object-based UFZ mapping method using very-high-resolution (VHR) remote sensing images. First, a multi-scale semantic segmentation network that achieves pixel-wised predictions is proposed to predict urban-functions for geographic objects by capturing multi-scale contextual information. Afterwards, a conditional random field (CRF) framework is designed to regroup objects into UFZs to produce the final UFZ map, wherein road vectors are incorporated to restrict the procedure. The presented object-as-analysis-unit scheme conquers the drawbacks of mapping-unit pre-determination and the semantic segmentation model provides accurate function information for objects, thus they can be applied for producing large-scale and fine-grained UFZ maps. In the experiment, the proposed method is evaluated by producing UFZ maps for Beijing and Shanghai, China, and competitive results with overall accuracy of 91.6% and 89.1% are achieved, respectively. Finally, the generated UFZ maps are utilized to analyze the urban-function structures of the two cities. The proposed method can be regarded as a significant development that appears to be promising and practical for mapping UFZ maps for real-world urban applications.

Land-Use/Land-Cover Change Detection Based on Class-Prior Object-Oriented Conditional Random Field Framework for High Spatial Resolution Remote Sensing Imagery

High spatial resolution (HSR) remote sensing images can reflect more subtle changes and more specific types of land use and land cover (LULC) due to the abundant spatial geometric information. In this article, a class-prior object-oriented conditional random field (COCRF) framework consisting of a binary change detection (CD) task and a multiclass CD task is proposed to fill the application gap. In the proposed framework, the class-prior knowledge is used to improve the construction of the unary potential in both the binary and multiclass CD tasks, to reduce the influence of spectral variability. The binary CD result provides a constraint to the multiclass CD result. As a result, both parts have effective interaction. The class posterior probability images of two dates can be obtained automatically with the class-prior knowledge by sample migration. Furthermore, an object constraint described by the class dispersion within the objects is added to improve the smoothness in local objects, while the pairwise potential improves the smoothness of the whole area by using the eight-neighborhood spectral information of the center pixel. By integrating the above approaches, the problems of error accumulation and the manual intervention required in the traditional multiclass CD methods can be relieved. An adaptive parameter estimation strategy is also adopted in the proposed framework, to save the time required for manual parameter setting. The proposed COCRF framework was validated on two HSR remote sensing image data sets, where it achieved a better performance than the other state-of-the-art CD methods.

Incorporating DeepLabv3+ and object-based image analysis for semantic segmentation of very high resolution remote sensing images

ABSTRACT Semantic segmentation of remote sensing images is an important but unsolved problem in the remote sensing society. Advanced image semantic segmentation models, such as DeepLabv3+, have achieved astonishing performance for semantically labeling very high resolution (VHR) remote sensing images. However, it is difficult for these models to capture the precise outlines of ground objects and explore the context information that revealing relationships among image objects for optimizing segmentation results. Consequently, this study proposes a semantic segmentation method for VHR images by incorporating deep learning semantic segmentation model (DeepLabv3+) and object-based image analysis (OBIA), wherein DSM is employed to provide geometric information to enhance the interpretation of VHR images. The proposed method first obtains two initial probabilistic labeling predictions using a DeepLabv3+ network on spectral image and a random forest (RF) classifier on hand-crafted features, respectively. These two predictions are then integrated by Dempster-Shafer (D-S) evidence theory to be fed into an object-constrained higher-order conditional random field (CRF) framework to estimate the final semantic labeling results with the consideration of the spatial contextual information. The proposed method is applied to the ISPRS 2D semantic labeling benchmark, and competitive overall accuracies of 90.6% and 85.0% are achieved for Vaihingen and Potsdam datasets, respectively.

SLCRF: Subspace Learning With Conditional Random Field for Hyperspectral Image Classification

Subspace learning (SL) plays an important role in hyperspectral image (HSI) classification, since it can provide an effective solution to reduce the redundant information in the image pixels of HSIs. Previous works about SL aim to improve the accuracy of HSI recognition. Using a large number of labeled samples, related methods can train the parameters of the proposed solutions to obtain better representations of HSI pixels. However, the data instances may not be sufficient enough to learn a precise model for HSI classification in real applications. Moreover, it is well-known that it takes much time, labor and human expertise to label HSI images. To avoid the aforementioned problems, a novel SL method that includes the probability assumption called subspace learning with conditional random field (SLCRF) is developed. In SLCRF, first, the 3D convolutional autoencoder (3DCAE) is introduced to remove the redundant information in HSI pixels. In addition, the relationships are also constructed using the spectral-spatial information among the adjacent pixels. Then, the conditional random field (CRF) framework can be constructed and further embedded into the HSI SL procedure with the semi-supervised approach. Through the linearized alternating direction method termed LADMAP, the objective function of SLCRF is optimized using a defined iterative algorithm. The proposed method is comprehensively evaluated using the challenging public HSI datasets. We can achieve stateof-the-art performance using these HSI sets.

Data-Augmented Hybrid Named Entity Recognition for Disaster Management by Transfer Learning

This research aims to build a Mandarin named entity recognition (NER) module using transfer learning to facilitate damage information gathering and analysis in disaster management. The hybrid NER approach proposed in this research includes three modules: (1) data augmentation, which constructs a concise data set for disaster management; (2) reference model, which utilizes the bidirectional long short-term memory–conditional random field framework to implement NER; and (3) the augmented model built by integrating the first two modules via cross-domain transfer with disparate label sets. Through the combination of established rules and learned sentence patterns, the hybrid approach performs well in NER tasks for disaster management and recognizes unfamiliar words successfully. This research applied the proposed NER module to disaster management. In the application, we favorably handled the NER tasks of our related work and achieved our desired outcomes. Through proper transfer, the results of this work can be extended to other fields and consequently bring valuable advantages in diverse applications.

Large-scale ALS Data Semantic Classification Integrating Location-Context-Semantics Cues by Higher-Order CRF.

We designed a location-context-semantics-based conditional random field (LCS-CRF) framework for the semantic classification of airborne laser scanning (ALS) point clouds. For ALS datasets of high spatial resolution but with severe noise pollutions, more contexture and semantics cues, besides location information, can be exploited to surmount the decrease of discrimination of features for classification. This paper mainly focuses on the semantic classification of ALS data using mixed location-context-semantics cues, which are integrated into a higher-order CRF framework by modeling the probabilistic potentials. The location cues modeled by the unary potentials can provide basic information for discriminating the various classes. The pairwise potentials consider the spatial contextual information by establishing the neighboring interactions between points to favor spatial smoothing. The semantics cues are explicitly encoded in the higher-order potentials. The higher-order potential operates at the clusters level with similar geometric and radiometric properties, guaranteeing the classification accuracy based on semantic rules. To demonstrate the performance of our approach, two standard benchmark datasets were utilized. Experiments show that our method achieves superior classification results with an overall accuracy of 83.1% on the Vaihingen Dataset and an overall accuracy of 94.3% on the Graphics and Media Lab (GML) Dataset A compared with other classification algorithms in the literature.

Integrating Dense LiDAR-Camera Road Detection Maps by a Multi-Modal CRF Model

Road detection is an important task in autonomous navigation systems. In this paper, we propose a road detection method via a LiDAR-camera fusion strategy to exploit both the range and color information. The whole system consists of three parts. In the LiDAR based part, we transform the discrete 3D LiDAR point clouds to continuous 2D LiDAR range images and propose a distance-aware height-difference based scanning approach to get the road estimations quickly. In the camera based part, we apply a light-weight transfer learning based road segmentation network. In the LiDAR-camera fusion part, we transform the detection results from LiDAR and camera to dense and binary ones to solve the data imbalance problem and fuse them in a multi-modal conditional random field (MM-CRF) framework. Experiments show that the proposed MM-CRF fusion method can operate in real-time and achieve competitive performance compared with the state-of-the-art road detection algorithms on the KITTI-Road benchmark.

Hybrid neural conditional random fields for multi-view sequence labeling

In traditional machine learning, conditional random fields (CRF) is the mainstream probability model for sequence labeling problems. CRF considers the relation between adjacent labels other than decoding each label independently, and better performance is expected to achieve. However, there are few multi-view learning methods involving CRF which can be directly used for sequence labeling tasks. In this paper, we propose a novel multi-view CRF model to label sequential data, called MVCRF, which well exploits two principles for multi-view learning: consensus and complementary. We first use different neural networks to extract features from multiple views. Then, considering the consistency among the different views, we introduce a joint representation space for the extracted features and minimize the distance between the two views for regularization. Meanwhile, following the complementary principle, the features of multiple views are integrated into the framework of CRF. We train MVCRF in an end-to-end fashion and evaluate it on two benchmark data sets. The experimental results illustrate that MVCRF obtains state-of-the-art performance: F1 score 95.44% for chunking on CoNLL-2000, 95.06% for chunking and 96.99% for named entity recognition (NER) on CoNLL-2003.

Chinese NER with Height-Limited Constituent Parsing

In this paper, we investigate how to improve Chinese named entity recognition (NER) by jointly modeling NER and constituent parsing, in the framework of neural conditional random fields (CRF). We reformulate the parsing task to heightlimited constituent parsing, by which the computational complexity can be significantly reduced, and the majority of phrase-level grammars are retained. Specifically, an unified model of neural semi-CRF and neural tree-CRF is proposed, which simultaneously conducts word segmentation, part-ofspeech (POS) tagging, NER, and parsing. The challenge comes from how to train and infer the joint model, which has not been solved previously. We design a dynamic programming algorithm for both training and inference, whose complexity is O(n·4h), where n is the sentence length and h is the height limit. In addition, we derive a pruning algorithm for the joint model, which further prunes 99.9% of the search space with 2% loss of the ground truth data. Experimental results on the OntoNotes 4.0 dataset have demonstrated that the proposed model outperforms the state-of-the-art method by 2.79 points in the F1-measure.

Self-Supervised Feature Learning With CRF Embedding for Hyperspectral Image Classification

The challenges in hyperspectral image (HSI) classification lie in the existence of noisy spectral information and lack of contextual information among pixels. Considering the three different levels in HSIs, i.e., subpixel, pixel, and superpixel, offer complementary information, we develop a novel HSI feature learning network (HSINet) to learn consistent features by self-supervision for HSI classification. HSINet contains a three-layer deep neural network and a multifeature convolutional neural network. It automatically extracts the features such as spatial, spectral, color, and boundary as well as context information. To boost the performance of self-supervised feature learning with the likelihood maximization, the conditional random field (CRF) framework is embedded into HSINet. The potential terms of unary, pairwise, and higher order in CRF are constructed by the corresponding subpixel, pixel, and superpixel. Furthermore, the feedback information derived from these terms are also fused into the different-level feature learning process, which makes the HSINet-CRF be a trainable end-to-end deep learning model with the back-propagation algorithm. Comprehensive evaluations are performed on three widely used HSI data sets and our method outperforms the state-of-the-art methods.

Semantic segmentation of multisensor remote sensing imagery with deep ConvNets and higher-order conditional random fields

Aerial images acquired by multiple sensors provide comprehensive and diverse information of materials and objects within a surveyed area. The current use of pretrained deep convolutional neural networks (DCNNs) is usually constrained to three-band images (i.e., RGB) obtained from a single optical sensor. Additional spectral bands from a multiple sensor setup introduce challenges for the use of DCNN. We fuse the RGB feature information obtained from a deep learning framework with light detection and ranging (LiDAR) features to obtain semantic labeling. Specifically, we propose a decision-level multisensor fusion technique for semantic labeling of the very-high-resolution optical imagery and LiDAR data. Our approach first obtains initial probabilistic predictions from two different sources: one from a pretrained neural network fine-tuned on a three-band optical image, and another from a probabilistic classifier trained on LiDAR data. These two predictions are then combined as the unary potential using a higher-order conditional random field (CRF) framework, which resolves fusion ambiguities by exploiting the spatial–contextual information. We utilize graph cut to efficiently infer the final semantic labeling for our proposed higher-order CRF framework. Experiments performed on three benchmarking multisensor datasets demonstrate the performance advantages of our proposed method.

Adaptive appearance separation for interactive image segmentation based on Dense CRF

Interactive segmentation has recently become a hot topic for its wide application. The authors propose an efficacious appearance separation model for interactive binary segmentation, which incorporates the difference of foreground and background colour models and the difference of corresponding geodesic models into the popular densely connected conditional random field (Dense CRF) framework. The proposed method can adaptively set relevant parameter values in this framework according to the characteristics of target images in a per-image manner, therefore, it gets rid of the dependence on specific datasets. After accomplishing a mean-field inference, the authors are able to get satisfactory results without the time-consuming parameter learning process and multiple iterative optimisations. Overall, the proposed approach is highly efficient and mitigates the contradiction between accuracy and segmentation efficiency. In addition, the proposed approach reduces the efforts of scribble-style interaction from users. The experimental results on three famous datasets show that the proposed method is superior to the other five new algorithms released in recent years regarding accuracy, and is faster than or close to them in runtime.

Region-Based Nonparametric Model for Interactive Image Segmentation

In this paper, we present a novel framework for interactive segmentation problems. It integrates a nonparametric model into the Conditional Random Field (CRF) framework, which can effectively combine high-level features with low-level features to represent image information. In the nonparametric model, multiple region layers are used to estimate data likelihood terms to overcome the bad regions generated by unsupervised methods. The likelihood values of each layer are calculated separately to reduce the computational cost. In addition, we analyze that the pixel layer has little effect on data estimation, so we remove it to further reduce the complexity of the algorithm. We employ the label consistency between pixels and their corresponding regions in smooth term estimation, which can be regard as a higher order potential for pixels. The data term and the smooth term are then performed together in Conditional Random Fields (CRFs) as a fine-tuning of the results. Experimental results show that the proposed method can segment images efficiently and accurately with fewer user inputs.

Gated CNN for visual quality assessment based on color perception

Visual quality assessment aims to build a computational model which can evaluate the quality of image with respect to human perception. As one of the important aspects to represent images, color provides rich information of images and deeply influences the visual perception. Thus, color harmony, which is defined as “two or more colors are sensed together as a single, pleasing, collective impression” (Holtzschue, 2011), is one of the important features to determine the visual/aesthetic quality of images. Despite the recent progress of color harmony models in respect to aesthetic quality assessment, most conventional approaches represent the color harmony only considering the distribution of co-occurrence colors but ignoring the spatial relationships between those neighbored colors. To overcome this limitation, we propose to take advantage of the intrinsic structural properties from conditional random field (CRF) to model the color harmony of images. In the CRF framework, we present a novel method that uses gated convolutional neural networks (CNNs) to calculate the probabilities of being high aesthetic quality for small patches and compute the harmony compatibilities between them, which can be considered as the associated and interactive potentials of CRF. Semantic tag of each image is also employed in our work to improve the proposed harmony model’s discriminant capability, which shows promising improvements for aesthetic quality assessment compared with existing color harmony models.

Automatic Building Rooftop Extraction From Aerial Images via Hierarchical RGB-D Priors

Accurate building rooftop extraction from high-resolution aerial images is of crucial importance in a wide range of applications. Owing to the varying appearance and large-scale range of scene objects, especially for building rooftops in different scales and heights, single-scale or individual prior-based extraction technique is insufficient in pursuing efficient, generic, and accurate extraction results. The trend toward integrating multiscale or several cue techniques appears to be the best way; thus, such integration is the focus of this paper. We first propose a novel salient rooftop detector integrating four correlative RGB-D priors (depth cue, uniqueness prior, shape prior, and transition surface prior) for improved rooftop extraction to address the preceding complex issues mentioned. Then, these correlative cues are computed from image layers created by our multilevel segmentation and further fused into the state-of-the-art high-order conditional random field (CRF) framework to locate the rooftop. Finally, an iterative optimization strategy is applied for high-quality solving, which can robustly handle varying appearance of building rooftops. Performance evaluations in the SZTAKI-INRIA benchmark data sets show that our method outperforms the traditional color-based algorithm and the original high-order CRF algorithm and its variants. The proposed algorithm is also evaluated and found to produce consistently satisfactory results for various large-scale, real-world data sets.

Fusion‐based holistic road scene understanding

This study addresses the problem of holistic road scene understanding based on the integration of visual and range data. To achieve the grand goal, the authors propose an approach that jointly tackles object-level image segmentation and semantic region labelling within a conditional random field (CRF) framework. Specifically, the authors first generate semantic object hypotheses by clustering 3D points, learning their prior appearance models, and using a deep learning method for reasoning their semantic categories. The learned priors, together with spatial and geometric contexts, are incorporated in CRF. With this formulation, visual and range data are fused thoroughly, and moreover, the coupled segmentation and semantic labelling problem can be inferred via graph cuts. The authors’ approach is validated on the challenging KITTI dataset that contains diverse complicated road scenarios. Both quantitative and qualitative evaluations demonstrate its effectiveness.

A supervised joint multi-layer segmentation framework for retinal optical coherence tomography images using conditional random field

Background and Objective:Accurate segmentation of the intra-retinal tissue layers in Optical Coherence Tomography (OCT) images plays an important role in the diagnosis and treatment of ocular diseases such as Age-Related Macular Degeneration (AMD) and Diabetic Macular Edema (DME). The existing energy minimization based methods employ multiple, manually handcrafted cost terms and often fail in the presence of pathologies. In this work, we eliminate the need to handcraft the energy by learning it from training images in an end-to-end manner. Our method can be easily adapted to pathologies by re-training it on an appropriate dataset.Methods:We propose a Conditional Random Field (CRF) framework for the joint multi-layer segmentation of OCT B-scans. The appearance of each retinal layer and boundary is modeled by two convolutional filter banks and the shape priors are modeled using Gaussian distributions. The total CRF energy is linearly parameterized to allow a joint, end-to-end training by employing the Structured Support Vector Machine formulation.Results:The proposed method outperformed three benchmark algorithms on four public datasets. The NORMAL-1 and NORMAL-2 datasets contain healthy OCT B-scans while the AMD-1 and DME-1 dataset contain B-scans of AMD and DME cases respectively. The proposed method achieved an average unsigned boundary localization error (U-BLE) of 1.52 pixels on NORMAL-1, 1.11 pixels on NORMAL-2 and 2.04 pixels on the combined NORMAL-1 and DME-1 dataset across the eight layer boundaries, outperforming the three benchmark methods in each case. The Dice coefficient was 0.87 on NORMAL-1, 0.89 on NORMAL-2 and 0.84 on the combined NORMAL-1 and DME-1 dataset across the seven retinal layers. On the combined NORMAL-1 and AMD-1 dataset, we achieved an average U-BLE of 1.86 pixels on the ILM, inner and outer RPE boundaries and a Dice of 0.98 for the ILM-RPEin region and 0.81 for the RPE layer.Conclusion:We have proposed a supervised CRF based method to jointly segment multiple tissue layers in OCT images. It can aid the ophthalmologists in the quantitative analysis of structural changes in the retinal tissue layers for clinical practice and large-scale clinical studies.

MPCaD: a multi-scale radiomics-driven framework for automated prostate cancer localization and detection

BackgroundQuantitative radiomic features provide a plethora of minable data extracted from multi-parametric magnetic resonance imaging (MP-MRI) which can be used for accurate detection and localization of prostate cancer. While most cancer detection algorithms utilize either voxel-based or region-based feature models, the complexity of prostate tumour phenotype in MP-MRI requires a more sophisticated framework to better leverage available data and exploit a priori knowledge in the field.MethodsIn this paper, we present MPCaD, a novel Multi-scale radiomics-driven framework for Prostate Cancer Detection and localization which leverages radiomic feature models at different scales as well as incorporates a priori knowledge of the field. Tumour candidate localization is first performed using a statistical texture distinctiveness strategy that leverages a voxel-resolution feature model to localize tumour candidate regions. Tumour region classification via a region-resolution feature model is then performed to identify tumour regions. Both voxel-resolution and region-resolution feature models are built upon and extracted from six different MP-MRI modalities. Finally, a conditional random field framework that is driven by voxel-resolution relative ADC features is used to further refine the localization of the tumour regions in the peripheral zone to improve the accuracy of the results.ResultsThe proposed framework is evaluated using clinical prostate MP-MRI data from 30 patients, and results demonstrate that the proposed framework exhibits enhanced separability of cancerous and healthy tissue, as well as outperforms individual quantitative radiomics models for prostate cancer detection.ConclusionQuantitative radiomic features extracted from MP-MRI of prostate can be utilized to detect and localize prostate cancer.