Hierarchical Conditional Random Field Research Articles

A Hierarchical Fusion SAR Image Change-Detection Method Based on HF-CRF Model

The mainstream methods for change detection in synthetic-aperture radar (SAR) images use difference images to define the initial change regions. However, methods can suffer from semantic collapse, which makes it difficult to determine semantic information about the changes. In this paper, we proposed a hierarchical fusion SAR image change-detection model based on hierarchical fusion conditional random field (HF-CRF). This model introduces multimodal difference images and constructs the fusion energy potential function using dynamic convolutional neural networks and sliding window entropy information. By using an iterative convergence process, the proposed method was able to accurately detect the change-detection regions. We designed a dynamic region convolutional semantic segmentation network with a two-branch structure (D-DRUNet) to accomplish feature fusion and the segmentation of multimodal difference images. The proposed network adopts a dual encoder–single decoder structure where the baseline is the UNet network that utilizes dynamic convolution kernels. D-DRUNet extracts multimodal difference features and completes semantic-level fusion. The Sobel operator is introduced to strengthen the multimodal difference-image boundary information and construct the dynamic fusion pairwise potential function, based on local boundary entropy. Finally, the final change result is stabilized by iterative convergence of the CRF energy potential function. Experimental results demonstrate that the proposed method outperforms existing methods in terms of the overall number of detection errors, and reduces the occurrence of false positives.

A hierarchical conditional random field-based attention mechanism approach for gastric histopathology image classification

In the Gastric Histopathology Image Classification (GHIC) tasks, which are usually weakly supervised learning missions, there is inevitably redundant information in the images. Therefore, designing networks that can focus on distinguishing features has become a popular research topic. In this paper, to accomplish the tasks of GHIC superiorly and assist pathologists in clinical diagnosis, an intelligent Hierarchical Conditional Random Field based Attention Mechanism (HCRF-AM) model is proposed. The HCRF-AM model consists of an Attention Mechanism (AM) module and an Image Classification (IC) module. In the AM module, an HCRF model is built to extract attention regions. In the IC module, a Convolutional Neural Network (CNN) model is trained with the attention regions selected, and then an algorithm called Classification Probability-based Ensemble Learning is applied to obtain the image-level results from the patch-level output of the CNN. In the experiment, a classification specificity of 96.67% is achieved on a gastric histopathology dataset with 700 images. Our HCRF-AM model demonstrates high classification performance and shows its effectiveness and future potential in the GHIC field. In addition, the AM module and transfer learning technique allow the network to generalize well to other types of image data except histopathology images, and we obtain 95.5% and 95.8% accuracies on IG02 and Oxford-IIIT Pet Datasets.

ASRNN: A recurrent neural network with an attention model for sequence labeling

Natural language processing (NLP) is useful for handling text and speech, and sequence labeling plays an important role by automatically analyzing a sequence (text) to assign category labels to each part. However, the performance of these conventional models depends greatly on hand-crafted features and task-specific knowledge, which is a time consuming task. Several conditional random fields (CRF)-based models for sequence labeling have been presented, but the major limitation is how to use neural networks for extracting useful representations for each unit or segment in the input sequence. In this paper, we propose an attention segmental recurrent neural network (ASRNN) that relies on a hierarchical attention neural semi-Markov conditional random fields (semi-CRF) model for the task of sequence labeling. Our model uses a hierarchical structure to incorporate character-level and word-level information and applies an attention mechanism to both levels. This enables our method to differentiate more important information from less important information when constructing the segmental representation. We evaluated our model on three sequence labeling tasks, including named entity recognition (NER), chunking, and reference parsing. Experimental results show that the proposed model benefited from the hierarchical structure, and it achieved competitive and robust performance on all three sequence labeling tasks.

Gastric histopathology image segmentation using a hierarchical conditional random field

For the Convolutional Neural Networks (CNNs) applied in the intelligent diagnosis of gastric cancer, existing methods mostly focus on individual characteristics or network frameworks without a policy to depict the integral information. Mainly, conditional random field (CRF), an efficient and stable algorithm for analyzing images containing complicated contents, can characterize spatial relation in images. In this paper, a novel hierarchical conditional random field (HCRF) based gastric histopathology image segmentation (GHIS) method is proposed, which can automatically localize abnormal (cancer) regions in gastric histopathology images obtained by an optical microscope to assist histopathologists in medical work. This HCRF model is built up with higher order potentials, including pixel-level and patch-level potentials, and graph-based post-processing is applied to further improve its segmentation performance. Especially, a CNN is trained to build up the pixel-level potentials and another three CNNs are fine-tuned to build up the patch-level potentials for sufficient spatial segmentation information. In the experiment, a hematoxylin and eosin (H&E) stained gastric histopathological dataset with 560 abnormal images are divided into training, validation and test sets with a ratio of 1:1:2. Finally, segmentation accuracy, recall and specificity of 78.91%, 65.59%, and 81.33% are achieved on the test set. Our HCRF model demonstrates high segmentation performance and shows its effectiveness and future potential in the GHIS field.

An Ego-Vision System for Discovering Human Joint Attention

Joint attention often happens during social interactions, in which individuals share focus on the same object. This article proposes an egocentric vision-based system (ego-vision system) that aims to discover the objects looked at jointly by a group of persons engaged in interactive activities. The proposed system relies on a collection of wearable eye-tracking cameras that provide an egocentric view of the interaction scenes as well as points-of-gaze measurement of each participant. Technically in our system, we develop a hierarchical conditional random field (CRF) based graphical model that can temporally localize joint attention periods and spatially segment objects of joint attention. By solving these two coupled tasks together in an iterative optimization procedure, we show that human joint attention can be reliably discovered from videos even with cluttered background and noisy gaze measurement. A new dataset of joint attention is collected and annotated for evaluating the two tasks of joint attention where two to four persons are involved. Experimental results demonstrate that our approach achieves state-of-the-art performance on both tasks of spatial segmentation and temporal localization of joint attention.

Hierarchical conditional random field model for multi‐object segmentation in gastric histopathology images

In this Letter, a hierarchical conditional random field (HCRF) model-based gastric histopathology image segmentation (GHIS) method is proposed, which can localise abnormal (cancer) regions in gastric histopathology images to assist histopathologists in medical work. First, to obtain pixel-level segmentation information, the authors retrain a convolutional neural network (CNN) to build up their pixel-level potentials. Then, to obtain abundant spatial segmentation information in patch level, they fine tune another three CNNs to build up their patch-level potentials. Thirdly, based on the pixel- and patch-level potentials, their HCRF model is structured. Finally, a graph-based post-processing is applied to further improve their segmentation performance. In the experiment, a segmentation accuracy of 78.91 % is achieved on a haematoxylin and eosin stained gastric histopathological dataset with 560 images, showing the effectiveness and future potential of the proposed GHIS method.

Multi-Scale Hierarchical CRF for Railway Electrification Asset Classification From Mobile Laser Scanning Data

A network of railway infrastructure is one of the most critical infrastructure assets for supporting the national economy and sustainable mobility. Safe and reliable maintenance of railway infrastructure is critical to ensure that rail systems run safely and punctually. Such maintenance requires regular surveying of railway assets, which typically relies on time-consuming and error-prone labor-centric visual inspection. In this paper, we propose a novel supervised method for automatically classifying electrification assets of railway networks using mobile laser scanning data. A hierarchical Conditional Random Field (CRF) was investigated in order to apply both smoothness constraint and spatial regularities, to improve the classification result made by local supervised classifiers. We use a multi-scale line representation of original data, which implicitly combines object geometry cues and makes computation efficient. Our approach focuses on learning the spatial regularities at multiple representation scales to thoroughly understand the railway electrification scene. The spatial regularities are formulated as relative spatial location in a middle range for different line primitive scales and relative displacement in a full range for the final coarsest line primitive scale. The experiment shows that learnt spatial regularities at full range with multi-scales can outperform the model with spatial regularities at limited local ranges.

Edge detection based on Improved Non-Maximum Suppression Method

Semantic Segmentation and edge detection are important research fields for scene understanding in computer vision. A hierarchical framework called Contextual Hierarchical Model (CHM) was proposed for semantic image segmentation and edge detection. It learned contextual information using a Logistic Disjunctive Normal Networks (LDNN) classifier. The class average accuracy of CHM was improved by defining a global constraint using Conditional Random Field (CRF), Hierarchical CRF (HCRF), Higher order HCRF (HHCRF). The LDNN was improved by using proximal gradient method which minimizes the quadratic error and it had high convergence rate than gradient descent method. The weight and bias terms of LDNN was optimized by using Grey Wolf Optimization algorithm (GWO) which improves the classification accuracy and it also reduces time complexity of LDNN. During the edge detection using CHM, a multi-scale strategy was adopted to compute edge maps. A Non-Maximum Suppression (NMS) was used to obtain the thinned edges in images. However, it is a post-processing step which consumes additional time in edge detection process. So, in this paper, the edge detection is interpreted as a classification problem where the thinned edges in images are obtained without any post-processing step. It reduces the time consumption. Two key ingredients such as loss and joint processing are included in NMS to improve the detection of thinned edges. The loss is used to penalize the double edge detection and the joint processing is used to reduce the loss of edge detection by including a pair features as additional feature for edge detection. The pair features, Haar, Histogram of Gradient (HOG) and SIFT features are given as input to LDNN to detect the edges that improve the efficiency of CHM based edge detection.

Post-Structuring Radiology Reports of Breast Cancer Patients for Clinical Quality Assurance.

Hospitals often set protocols based on well defined standards to maintain the quality of patient reports. To ensure that the clinicians conform to the protocols, quality assurance of these reports is needed. Patient reports are currently written in free-text format, which complicates the task of quality assurance. In this paper, we present a machine learning based natural language processing system for automatic quality assurance of radiology reports on breast cancer. This is achieved in three steps: we i) identify the top-level structure (headings) of the report, ii) classify the report content into the top-level headings, and iii) convert the free-text detailed findings in the report to a semi-structured format (post-structuring). Top level structure and content of report were predicted with an F1 score of 0.97 and 0.94, respectively, using Support Vector Machine (SVM) classifiers. For automatic structuring, our proposed hierarchical Conditional Random Field (CRF) outperformed the baseline CRF with an F1 score of 0.78 versus 0.71. The determined structure of the report is represented in semi-structured XML format of the free-text report, which helps to easily visualize the conformance of the findings to the protocols. This format also allows easy extraction of specific information for other purposes such as search, evaluation, and research.

Weakly Supervised Segmentation of SAR Imagery Using Superpixel and Hierarchically Adversarial CRF

Synthetic aperture radar (SAR) image segmentation aims at generating homogeneous regions from a pixel-based image and is the basis of image interpretation. However, most of the existing segmentation methods usually neglect the appearance and spatial consistency during feature extraction and also require a large number of training data. In addition, pixel-based processing cannot meet the real time requirement. We hereby present a weakly supervised algorithm to perform the task of segmentation for high-resolution SAR images. For effective segmentation, the input image is first over-segmented into a set of primitive superpixels. This algorithm combines hierarchical conditional generative adversarial nets (CGAN) and conditional random fields (CRF). The CGAN-based networks can leverage abundant unlabeled data learning parameters, reducing their reliance on the labeled samples. In order to preserve neighborhood consistency in the feature extraction stage, the hierarchical CGAN is composed of two sub-networks, which are employed to extract the information of the central superpixels and the corresponding background superpixels, respectively. Afterwards, CRF is utilized to perform label optimization using the concatenated features. Quantified experiments on an airborne SAR image dataset prove that the proposed method can effectively learn feature representations and achieve competitive accuracy to the state-of-the-art segmentation approaches. More specifically, our algorithm has a higher Cohen’s kappa coefficient and overall accuracy. Its computation time is less than the current mainstream pixel-level semantic segmentation networks.

Semantic image segmentation using an improved hierarchical graphical model

Hierarchical graphical models can incorporate jointly several tasks in a unified framework. By applying this approach, information exchange among tasks would improve the results. A hierarchical conditional random field (CRF) is proposed here to improve the semantic image segmentation. Although this newly proposed model applies the information of several tasks, its run time is comparable with the contemporary approaches. This method is evaluated on MSRC dataset and has shown similar or better segmentation accuracy in comparison with models where CRFs or hierarchical models are adopted.

Simultaneous Volumetric Segmentation of Vertebral Bodies and Intervertebral Discs on Fat-Water MR Images.

Fat-water magnetic resonance (MR) images allow automated noninvasive analysis of morphological properties and fat fractions of vertebral bodies (VBs) and intervertebral discs (IVDs) that constitute an important part of human biomechanical systems. In this paper, we propose a fully automated approach for simultaneously segmenting multiple VBs and IVDs on fat-water MR images without prior localization or geometry estimation. This method involved a hierarchical random forest (HRF) classifier and a hierarchical conditional random field (HCRF) that encoded a multi-resolution image pyramid based on a set of multiscale local and contextual features. The HRF classifier employed penalized multivariate linear discriminants and SMOTEBagging to handle limited and imbalanced training data with large feature dimension. The HCRF estimated optimum labels according to their spatial and hierarchical consistencies by using the layer-wise significant features determined over the trained HRF classifier. To handle variable sample numbers at different resolutions, resolution-specific hyperparameters were used. This method was trained and evaluated for segmenting 15 thoracic and lumbar VBs and their IVDs on fat-water MR images of a subset of a large cohort data set. It was further evaluated for segmenting seven IVDs of the lower spine on fat-water images of a public grand challenge. These evaluations revealed the comparable accuracy of this method with the state-of-the-art while requiring less computational burden due to a simultaneous localization and segmentation.

High-Resolution Remote Sensing Image Classification Using Associative Hierarchical CRF Considering Segmentation Quality

This letter proposes an associative hierarchical conditional random field (AHCRF) model to improve the classification accuracy of high-resolution remote sensing images. It considers segmentation quality of superpixels, avoids a time-consuming selection of optimal scale parameters, and alleviates the problem of classification accuracy sensitive to undersegmentation errors that is present in traditional object-oriented classification methods. The model is built on a graph hierarchy, including the pixel layer as a base layer and multiple superpixel layers derived from a mean shift presegmentation. It extracts clustered features of pixels for superpixels at each layer and then defines the potentials of the AHCRF model. We suggest a weighted version of the interlayer potential using the size of a superpixel as a measure to reflect segmentation quality. In this way, erroneously labeled pixels of a superpixel are penalized. Experiments are presented using a part of the downsampled Vaihingen data from the ISPRS benchmark data set. Results confirm that our model shows more than 80% overall classification accuracy and is superior to the original AHCRF model and comparable to other models. It also alleviates the choosing of suitable segmentation parameters.

A hierarchical model for recognizing alarming states in a batteryless sensor alarm intervention for preventing falls in older people

Falls are common among older people, especially in hospitals and nursing homes. The combination of pervasive sensing and statistical learning methods is creating new possibilities for automatic monitoring of activities of hospitalized older people to provide targeted and timely supervision by clinical staff to reduce falls. In this paper we introduce a hierarchical conditional random fields model to predict alarming states (being out of the bed or chair) from a passive wearable embodiment of a sensor worn over garment to provide an intervention mechanism to reduce falls. Our approach predicts alarm states in real time and avoids the use of empirically determined heuristics methods alone or in combination with machine learning based models, or multiple cascaded classifiers for generating alarms from activity prediction streams. Instead, the proposed hierarchical approach predicts alarms based on learned relationships between alarms, sensor information and predicted low-level activities. We evaluate the performance of the approach with 14 healthy older people and 26 hospitalized older patients and demonstrate similar or better performance than machine learning based approaches combined with heuristics based methods.

Hierarchical semi-Markov conditional random fields for deep recursive sequential data

We present the hierarchical semi-Markov conditional random field (HSCRF), a generalisation of linear-chain conditional random fields to model deep nested Markov processes. It is parameterised as a conditional log-linear model and has polynomial time algorithms for learning and inference. We derive algorithms for partially-supervised learning and constrained inference. We develop numerical scaling procedures that handle the overflow problem. We show that when depth is two, the HSCRF can be reduced to the semi-Markov conditional random fields. Finally, we demonstrate the HSCRF on two applications: (i) recognising human activities of daily living (ADLs) from indoor surveillance cameras, and (ii) noun-phrase chunking. The HSCRF is capable of learning rich hierarchical models with reasonable accuracy in both fully and partially observed data cases.

Dynamic Gesture Recognition based on Image Sequence

This paper proposed an algorithm for 3D hands tracking on the learned hierarchical latent variable space, which employs a Hierarchical Gaussian Process Latent Variable Model(HGPLVM) to learn the hierarchical latent space of hands motion and the nonlinear mapping from the hierarchical latent space to the pose space simultaneously. Nonlinear mappings from the hierarchical latent space to the space of hand images are constructed using radial basis function interpolation method. With these mappings, particles can be projected into hand images and measured in the image space directly. Particle filters with fewer particles are used to track the hand on the learned hierarchical low-dimensional space. Then the Hierarchical Conditional Random Field, which can capture extrinsic class dynamics and learn the relationship between motions of hand parts and different hand gestures simultaneously, is presented to model the continuous hand gestures. Experimental results show that our proposed method can track articulated hand robustly and approving recognition performance has also been achieved on the user-defined hand gesture dataset.

Hierarchical Conditional Random Fields Model for Semisupervised SAR Image Segmentation

The conditional random field (CRF) model is suitable for the image segmentation because this model relaxes the assumption of conditional independence of the observed data and models the data-dependent label interaction in the image modeling. However, this model has a limited ability to capture the global and local image information from the perspective of multiresolution analysis. Moreover, for synthetic aperture radar (SAR) image segmentation, SAR scattering statistics that are essential to SAR image processing are not considered in the CRF model. In this paper, we propose a hierarchical CRF (HIECRF) model for SAR image segmentation. The HIECRF model belongs to the discriminative models according to the semantic structure. While inheriting the advantages of the CRF model, the HIECRF model achieves the integration of the image features and SAR scattering statistics and captures the contextual structure information in the spatial and scale spaces. Moreover, we derive a hierarchical inference algorithm for the HIECRF model in virtue of the mean-field approximation (MFA) to provide the maximization of the posterior marginal (MPM) estimate of the HIECRF model. Then, by the bottom-up and the top-down recursions in the hierarchical inference procedure, the HIECRF model effectively exploits the global and local image information, including the contextual structures, the image features, and the scattering statistics, to achieve the MPM segmentation. The effectiveness of the HIECRF model is demonstrated by the application to the semisupervised segmentation of the simulated images and the real SAR images.

Improved hierarchical conditional random field model for object segmentation

Although the hierarchical conditional random field (HCRF) model has been successfully applied to multi-class object segmentation, there is still room for improvement. Firstly, the pairwise potential in the HCRF model has the tendency to over-smooth boundaries of regions that are similar to their neighbors. Secondly, the higher-order potential associated with multiple unsupervised segments is prone to producing incorrect guidance to inference in the under-segmentation situation. Finally, the co-occurrence potential as a measure of inter-object relationships cannot completely suppress some uncommon combinations of object classes due to joint optimization of multi-potential cost function. To alleviate these problems, we propose an improved HCRF model that efficiently combines information from global, middle and local scales for object segmentation in this paper. At the global scale, scene categorization technique is adopted to recognize the scene category of an image. The scene consistency then enforces object segmentation to align with feasible labels in specific scenes at the local and middle scales. Furthermore, an improved pairwise potential and a segment-reliable consistency potential are developed at the local and middle scales, respectively. These potentials rectify the over-smoothness issues by propagating the believed labeling from the unary potential and perform coherent inference by ensuring reliable segment consistency. Experimental results on the MSRC-21 dataset demonstrate that the improved HCRF model achieves better subjective results, as well as state-of-the-art objective results in terms of both global accuracy of 87.98 % and average accuracy of 81.43 %.

Accurate object retrieval for high-resolution remote-sensing imagery using high-order topic consistency potentials

We propose incorporating semantic topic information into a hierarchical conditional random fields (CRFs) framework to promote object recognition and retrieval accuracy. Specially, we devise convenient yet effective methods based on multiple segmentations to perform accurate image retrieval tasks for rigid and amorphous man-made objects. Through a robust topic consistency potential (RTCP) modelling approach, we perform accurate multi-class segmentation on high-resolution remote-sensing images. The generated segments can be readily used for object recognition and discovery. We report satisfactory the performance on two sets of high-resolution remote-sensing images that cover a highly populated urban area and a rural area, respectively. Experimental results demonstrate that our approach outperforms the state-of-the-art CRF models, due to its ability to capture inherent semantic information for efficient object recognition and boundary discovery.

Anatomy packing with hierarchical segments: an algorithm for segmentation of pulmonary nodules in CT images.

BackgroundThis paper proposes a semantic segmentation algorithm that provides the spatial distribution patterns of pulmonary ground-glass nodules with solid portions in computed tomography (CT) images.MethodsThe proposed segmentation algorithm, anatomy packing with hierarchical segments (APHS), performs pulmonary nodule segmentation and quantification in CT images. In particular, the APHS algorithm consists of two essential processes: hierarchical segmentation tree construction and anatomy packing. It constructs the hierarchical segmentation tree based on region attributes and local contour cues along the region boundaries. Each node of the tree corresponds to the soft boundary associated with a family of nested segmentations through different scales applied by a hierarchical segmentation operator that is used to decompose the image in a structurally coherent manner. The anatomy packing process detects and localizes individual object instances by optimizing a hierarchical conditional random field model. Ninety-two histopathologically confirmed pulmonary nodules were used to evaluate the performance of the proposed APHS algorithm. Further, a comparative study was conducted with two conventional multi-label image segmentation algorithms based on four assessment metrics: the modified Williams index, percentage statistic, overlapping ratio, and difference ratio.ResultsUnder the same framework, the proposed APHS algorithm was applied to two clinical applications: multi-label segmentation of nodules with a solid portion and surrounding tissues and pulmonary nodule segmentation. The results obtained indicate that the APHS-generated boundaries are comparable to manual delineations with a modified Williams index of 1.013. Further, the resulting segmentation of the APHS algorithm is also better than that achieved by two conventional multi-label image segmentation algorithms.ConclusionsThe proposed two-level hierarchical segmentation algorithm effectively labelled the pulmonary nodule and its surrounding anatomic structures in lung CT images. This suggests that the generated multi-label structures can potentially serve as the basis for developing related clinical applications.