Higher-order Markov Random Fields Research Articles

High-Order Markov Random Field as Attention Network for High-Resolution Remote-Sensing Image Compression

Content-weighted compression scheme for high-resolution remote-sensing (RS) images can be well modeled by Markov random field (MRF)-oriented attention. This article addresses high-resolution RS image compression by incorporating MRF into attention mechanism. To this end, we reformulate the attention mechanism with MRF-based probabilistic graph modeling implicitly and combine the target of image compression and parameter learning of MRF in a unified framework, namely high-order MRF-oriented attention (HMA) network. Specifically, HMA extends key-value query (KVQ) pairwise terms of the vanilla attention to high-order terms, by which the prior information could be expressed effectively to boost performance of high-resolution RS image compression. It is noted that several superiorities of HMA are listed. First, unlike the vanilla attention network that apt to yield coarse features, HMA is capable of output more pleasing decoding results. Second, HMA can accelerate the convergence in the training of the deep neural networks (DNNs), thus facilitating deploying it on resource-limited IOT devices. Third, HMA demonstrates its potential of processing semantic joint task. Moreover, We thoroughly evaluate our approach on standard data sets of varying resolutions, the proposed framework performs favorably against most image coding standards and DNN-based codecs on the ISPRS Vaihingen data set and the USC-SIPI data set especially at low bit rates.

EVALUATION OF DEBLOCKING METHODS FOR CHANG’E-4 DESCENT IMAGES

Abstract. Chang’e-4 lunar probe has successfully landed on the far side of the moon in Von Kármán crater inside the South Pole-Aitken (SPA) basin at 10:26 am on January 3, 2019. Due to the reduction of the coding rate, obvious block effects appear at the boundaries of descent images. Unblock, adaptive fast bilateral filtering, structure-texture enhancement and high-order Markov random field methods, are applied to remove the block effect of the descent images. Based on analysing the quality of descent images, quantitative comparison of four methods is performed using simulated compressed 1:64 descent images and real images. Comprehensive analysis was performed using typical measures such as PSNR, SSIM and NIQE. Experimental results show that adaptive fast bilateral filtering is better than other methods. The deblocked 1:64 image sequences have been used to assist localizing the landing point quickly during the mission.

Adaptive Thouless–Anderson–Palmer Equation for Higher-order Markov Random Fields

The adaptive Thouless--Anderson--Palmer (TAP) mean-field approximation is one of the advanced mean-field approaches, and it is known as a powerful accurate method for Markov random fields (MRFs) with quadratic interactions (pairwise MRFs). In this study, an extension of the adaptive TAP approximation for MRFs with many-body interactions (higher-order MRFs) is developed. We show that the adaptive TAP equation for pairwise MRFs is derived by naive mean-field approximation with diagonal consistency. Based on the equivalence of the approximate equation obtained from the naive mean-field approximation with diagonal consistency and the adaptive TAP equation in pairwise MRFs, we formulate approximate equations for higher-order Boltzmann machines, which is one of simplest higher-order MRFs, via the naive mean-field approximation with diagonal consistency.

Lung 4D CT Image Registration Based on High-Order Markov Random Field.

To solve the problem that traditional image registration methods based on continuous optimization for large motion lung 4D CT image sequences are easy to fall into local optimal solutions and lead to serious misregistration, a novel image registration method based on high-order Markov Random Field (MRF) is proposed. By analyzing the effect of the deformation field constraint of the potential functions with different order cliques in MRF model, energy functions with high-order cliques form are designed separately for 2D and 3D images to preserve topology of the deformation field. In order to preserve the topology of the deformation field more effectively, it is necessary to apply a smooth term and a topology preservation term simultaneously in the energy function and use logarithmic function to impose a penalty on the Jacobian matrix with high-order cliques in the topology preservation term. For the complexity of the designed energy function with high-order cliques form, Markov Chain Monte Carlo (MCMC) algorithm is used to solve the optimization problem of the designed energy function. To address the high computational requirements in lung 4D CT image registration, a multi-level processing strategy is adopted to reduce the space complexity of the proposed registration method and promotes the computational efficiency. In the DIR-lab dataset with 4D CT images and the COPD (Chronic Obstructive Pulmonary Disease) dataset with 3D CT images, the average target registration error (TRE) of our proposed method can reach 0.95 mm respectively.

A Hybrid Model Integrating Spatial Pattern, Spatial Correlation, and Edge Information for Image Classification

This paper develops a novel hybrid model that integrates three spatial contexts into probabilistic classifiers for remote sensing classification. First, spatial pattern is introduced using multiple-point geostatistics (MPGs) to characterize the general distribution and arrangement of land covers. Second, spatial correlation is incorporated using spatial covariance to quantify the dependence between pixels. Third, an edge-preserving filter based on the Sobel mask is introduced to avoid the over-smoothing problem. These three types of contexts are combined with the spectral information from the original image within a higher-order Markov random field (MRF) framework for classification. The developed model is capable of classifying complex and diverse land cover types by allowing effective anisotropic filtering of the image while retaining details near edges. Experiments with three remote sensing images from different sources based on three probabilistic classifiers obtained results that significantly improved classification accuracies when compared with other popular contextual classifiers and most state-of-the-art methods.

Precise Markov random field model-based phase unwrapping method for airborne interferometric synthetic aperture radar imaging

Markovian model approach with graph cuts is effectively applied for phase unwrapping (PU) in interferometric synthetic aperture radar (InSAR) domain particularly for data with big noises and discontinuities. The first-order prior model in Markov random field (MRF) is widely used in Markovian model approach, which is able to solve PU problem in homogeneous scenarios. However, the first-order MRF method cannot interpret local details well in abundant patterns of interferograms under the requirement of high accuracy due to the limitation of first-order prior information. Hence, the conventional first-order model is no longer applicable in this situation. In terms of this problem, the shortcoming of the first-order model is analyzed, and a precise second-order MRF approach is proposed in this paper. Furthermore, a graph cuts-based optimization PU algorithm is designed under a high-order MRF mode, which improves the PU accuracy in detail areas. Without apparent increase of computational complexity, the proposed method is more robust compared with the conventional first-order MRF-based methods. A set of experimental results on both simulation and real measured millimeter-wave InSAR data illustrate the effectiveness of the proposal.

INDOOR SEMANTIC SEGMENTATION FROM RGB-D IMAGES BY INTEGRATING FULLY CONVOLUTIONAL NETWORK WITH HIGHER-ORDER MARKOV RANDOM FIELD

Abstract. Indoor scenes have the characteristics of abundant semantic categories, illumination changes, occlusions and overlaps among objects, which poses great challenges for indoor semantic segmentation. Therefore, we in this paper develop a method based on higher-order Markov random field model for indoor semantic segmentation from RGB-D images. Instead of directly using RGB-D images, we first train and perform RefineNet model only using RGB information for generating the high-level semantic information. Then, the spatial location relationship from depth channel and the spectral information from color channels are integrated as a prior for a marker-controlled watershed algorithm to obtain the robust and accurate visual homogenous regions. Finally, higher-order Markov random field model encodes the short-range context among the adjacent pixels and the long-range context within each visual homogenous region for refining the semantic segmentations. To evaluate the effectiveness and robustness of the proposed method, experiments were conducted on the public SUN RGB-D dataset. Experimental results indicate that compared with using RGB information alone, the proposed method remarkably improves the semantic segmentation results, especially at object boundaries.

Semantic Labeling of Mobile LiDAR Point Clouds via Active Learning and Higher Order MRF

Using mobile Light Detection and Ranging point clouds to accomplish road scene labeling tasks shows promise for a variety of applications. Most existing methods for semantic labeling of point clouds require a huge number of fully supervised point cloud scenes, where each point needs to be manually annotated with a specific category. Manually annotating each point in point cloud scenes is labor intensive and hinders practical usage of those methods. To alleviate such a huge burden of manual annotation, in this paper, we introduce an active learning method that avoids annotating the whole point cloud scenes by iteratively annotating a small portion of unlabeled supervoxels and creating a minimal manually annotated training set. In order to avoid the biased sampling existing in traditional active learning methods, a neighbor-consistency prior is exploited to select the potentially misclassified samples into the training set to improve the accuracy of the statistical model. Furthermore, lots of methods only consider short-range contextual information to conduct semantic labeling tasks, but ignore the long-range contexts among local variables. In this paper, we use a higher order Markov random field model to take into account more contexts for refining the labeling results, despite of lacking fully supervised scenes. Evaluations on three data sets show that our proposed framework achieves a high accuracy in labeling point clouds although only a small portion of labels is provided. Moreover, comparative experiments demonstrate that our proposed framework is superior to traditional sampling methods and exhibits comparable performance to those fully supervised models.

Learning scene-aware image priors with high-order Markov random fields

Many methods have been proposed to learn image priors from natural images for the ill-posed image restoration tasks. However, many prior learning algorithms assume that a general prior distribution is suitable for over all kinds of images. Since the contents of the natural images and the corresponding low-level statistical characteristics vary from scene to scene, we argue that learning a universal generative prior for all natural images may be imperfect. Although the universal generative prior can remove artifacts and reserve a natural smoothness in image restoration, it also tends to introduce unreal flatness and clutter textures. To address this issue, in this paper, we present to learn a scene-aware image prior based on the high-order Markov random field (MRF) model (SA-MRF). With this model, we jointly learn a set of shared low-level features and different potentials for specific scene contents. In prediction, a good prior can be adapted to the given degenerated image with the scene content perception. Experimental results on the image denoising and inpainting tasks demonstrate the efficiency of the SA-MRF on both numerical evaluation and visual compression.

NMPE: A normalized metric for measuring generalized spatial distortion of multispectral panshapening fusion

Adaptivity is important in remote sensing image fusion because of the data-intensive and mass-driven processing platforms that call for reliable evaluation metrics to assess the runtime fusion procedure. Spatial distortion, including poor detail, visual disorder, or over-injection, has not been measured as effectively as in the spectral domain. A new metric, namely normalized mean potential energy (NMPE), is proposed in this paper to check the generalized spatial distortion of fused images by calculating the potential energy of marginal filtering distributions using information of high-order Markov random fields. NMPE is defined based on the GFoE model, which is a new high-order model that we built for remote sensing image applications. To incorporate the evaluation experience of human vision system into the GFoE model, the real zero-mean Gabor filters with multiple directions and scales are used as feature extractors, and the Gaussian scale mixture model as the expert function. The model parameters are trained from 200 images of the Berkeley segmentation dataset. Poor detail in a fused image tends to result in small NMPE evaluation with respect to small Gabor scales. Our observation shows that scale invariance exists only for “good” details, so we use large scales of Gabor functions to detect visual disorder. Over-injection is checked when the fused NMPE is much higher than 1. In the experimental procedure, satellite images from Quickbird, LandSat-7, and SPOT-5 were put to fusion with five popular methods to produce different images for visual and digital comparison. It can be concluded from the experiment that NMPE is in line with our subjective judgment to measure the pansharpening quality in terms of enhanced detail, visual distortion, and over-injection.

Variational JPEG artifacts suppression based on high-order MRFs

The Block-wise Discrete Cosine Transform (B-DCT) based compression technique has been widely used in image and video coding standards. However, at high compression ratios, the coded images inevitably contain annoying blocking artifacts. In this paper, the author proposes a novel variational model for blocking artifacts suppression, which combines a discriminatively trained Fields of Experts (FoE) image prior model and the indicator function of the quantization constraint set (QCS). The FoE prior model is a filter-based higher-order Markov Random Fields (MRF) model, and it has proven to be effective for many image restoration problems. The resulting variational model leads to a generally difficult non-convex optimization problem, which can be efficiently solved by a recently proposed non-convex optimization algorithm. Numerical experiments show that the proposed deblocking approach leads to visually strongly comparable performance to state-of-the-art deblocking methods across a range of compression levels. Furthermore, our method can achieve higher PSNR-B results, which is a block-sensitive index, specialized for deblocked image evaluation and correlates well with subjective quality. Besides, the proposed model comes along with the additional advantage of high efficiency.

Higher-Order Graph Principles towards Non-Rigid Surface Registration.

This paper casts surface registration as the problem of finding a set of discrete correspondences through the minimization of an energy function, which is composed of geometric and appearance matching costs, as well as higher-order deformation priors. Two higher-order graph-based formulations are proposed under different deformation assumptions. The first formulation encodes isometric deformations using conformal geometry in a higher-order graph matching problem, which is solved through dual-decomposition and is able to handle partial matching. Despite the isometry assumption, this approach is able to robustly match sparse feature point sets on surfaces undergoing highly anisometric deformations. Nevertheless, its performance degrades significantly when addressing anisometric registration for a set of densely sampled points. This issue is rigorously addressed subsequently through a novel deformation model that is able to handle arbitrary diffeomorphisms between two surfaces. Such a deformation model is introduced into a higher-order Markov Random Field for dense surface registration, and is inferred using a new parallel and memory efficient algorithm. To deal with the prohibitive search space, we also design an efficient way to select a number of matching candidates for each point of the source surface based on the matching results of a sparse set of points. A series of experiments demonstrate the accuracy and the efficiency of the proposed framework, notably in challenging cases of large and/or anisometric deformations, or surfaces that are partially occluded.

Insights into prior learning for despeckling SAR images

This study presents an exploratory comparative study for learning the priors of synthetic aperture radar (SAR) images, using the Fields of Experts approach, a filter-based higher-order Markov random fields model, which has substantially improved the learning capability of the entire image priors. First, the authors provide new insights into prior learning for despeckling SAR images, which commonly exhibits a variety of the likelihood functions due to the complex physical process of the scattering. Second, the authors introduce a prior learning framework using a bi-level optimisation algorithm. Third, some interesting experiments are conducted to learn the priors of SAR images. Finally, the authors validate the learned priors on despeckling SAR images. It suggests that the representation of the prior is more accurate by using the training samples from the filtered real SAR images themselves than from both the optical images of the Earth's surface and the natural images.

Denoising model for parallel magnetic resonance imaging images using higher‐order Markov random fields

This study presents a novel robust method for Bayesian denoising of parallel magnetic resonance imaging (pMRI) images. For the first time, the authors’ proposal applies fields of experts (FoE), a filter-based higher-order Markov random field (MRF), to model the prior of the pMRI image statistics. The noise in pMRI data behaves to be non-central Chi (nc-χ) distributed. In practice, correlation between coils exists, resulting in that nc-χ distribution does not hold anymore and the spatially varying noise problem. Thus, preservation of fine textures requires to adapt locally the estimation. Therefore, more precisely, the noise is reduced by using a sliding window scheme. In each window, the likelihood probability function is accurately modelled from corrupted data by using an innovative Gaussian mixture model (GMM). The parameters of GMM are calculated by applying an iterative expectation maximisation approach. With the priors via the learned FoE model and the likelihood function via GMM, a maximum a posteriori (MAP) estimator is formulated. Then, the noise in the each window is filtered by applying an efficient non-linear quasi-Newton method to explore an optimal solution for the MAP estimator. Finally, experiments have been conducted on both the simulated and real data to compare the proposed model with some state-of-the-art denoising methods. The experimental results demonstrate the robustness and effectiveness of the proposed denoising model.

An Efficient Variational Method for Restoring Images with Combined Additive and Multiplicative Noise

This paper proposes a novel variational model to remove either independent additive or multiplicative noise from synthetic and natural digital images via the fractional-order derivative operator. The non-local characteristics of fractional derivatives can help preserve textures and eliminate the “blocky effect”. The proposed strategy uses the fractional-order total variation (FOTV)-norm, combined with the fields of experts-image prior model, a filter-based higher-order Markov Random Fields (MRF) method which is effective for image restoration. The present model combines advantages of both FOTV and higher order MRF and results in good restoration. In this study, a fast alternating minimization algorithm is also employed to solve minimization problem. Compared with the other well-established methods, experimental results show the effectiveness of the proposed method for de-noising images contaminated by combined additive and multiplicative noises. In addition, we also discuss parameter dependency and computational analysis in details.

Incorporating higher-order point distribution model priors into MRFs using convex quadratic programming

Recently, with the advent of powerful optimisation algorithms for Markov random fields (MRFs), priors of high arity (more than two) have been put into practice more widely. The statistical relationship between object parts encoding shape in a covariant space, also known as the point distribution model (PDM), is a widely employed technique in computer vision which has been largely overlooked in the context of higher-order MRF models. This paper focuses on such higher-order statistical shape priors and illustrates that in a spatial transformation invariant space, these models can be formulated as convex quadratic programmes. As such, the associated energy of a PDM may be optimised efficiently using a variety of different dedicated algorithms. Moreover, it is shown that such an approach in the context of graph matching can be utilised to incorporate both a global rigid and a non-rigid deformation prior into the problem in a parametric form, a problem which has been rarely addressed in the literature. The paper then illustrates an application of PDM priors for different tasks using graphical models incorporating factors of different cardinalities.

Higher‐order MRFs based image super resolution: why not MAP?

A trainable filter-based higher-order Markov random fields model – the so called fields of experts (FoE), has proved a highly effective image prior model for many classic image restoration problems. Generally, two options are available to incorporate the learned FoE prior in the inference procedure: (i) sampling-based minimum mean square error (MMSE) estimate, and (ii) energy minimisation-based maximum a posteriori (MAP) estimate. This study is devoted to the FoE prior based single image super resolution (SR) problem, and the author suggest to make use of the MAP estimate for inference based on two facts: (i) It is well-known that the MAP inference has a remarkable advantage of high computational efficiency, while the sampling-based MMSE estimate is very time consuming. (ii) Practical SR experiment results demonstrate that the MAP estimate works equally well compared with the MMSE estimate with exactly the same FoE prior model. Moreover, it can lead to even further improvements by incorporating the discriminatively trained FoE prior model. In summary, the author hold that for higher-order natural image prior based SR problem, it is better to employ the MAP estimate for inference.

Normalized filter pool for prior modeling of nature images

Markov random field (MRF), as one of special undirected graphs, is widely used in modeling priors of natural images. Targeting to learn better prior models from a given database, we explore the natural image statistics at different scales and build normalized filter pool, a kind of high-order MRF, for prior learning of nature images. The main contribution of the proposed model is that we construct a multi-scale MRF model through constraining the norms of filters in kernel space and integrate all the filtering responses in a unified framework. We formulate both learning and inference as constrained optimization problems and solve them using augmented Lagrange method. The experiment results demonstrate that the normalization of filters at different scales helps to achieve fast convergence in learning stage and obtain superior performance in image restoration, e.g., image denoising and image inpainting.

Poisson Noise Reduction with Higher-Order Natural Image Prior Model

Poisson denoising is an essential issue for various imaging applications, such as night vision, medical imaging, and microscopy. State-of-the-art approaches are clearly dominated by patch-based non-local methods in recent years. In this paper, we aim to propose a local Poisson denoising model with both structural simplicity and good performance. To this end, we consider a variational modeling to integrate the so-called fields of experts (FoE) image prior, that has proven an effective higher-order Markov random fields model for many classic image restoration problems. We exploit several feasible variational variants for this task. We start with a direct modeling in the original image domain by taking into account the Poisson noise statistics, which performs generally well for the cases of high signal-to-noise ratio (SNR). However, this strategy encounters problem in cases of low SNR. Then we turn to an alternative modeling strategy by using the Anscombe transform and Gaussian statistics derived data term. We retrain the FoE prior model directly in the transform domain. With the newly trained FoE model, we end up with a local variational model providing strongly competitive results against state-of-the-art nonlocal approaches, meanwhile bearing the property of simple structure. Furthermore, our proposed model comes along with an additional advantage, that the inference is very efficient as it is well suited for parallel computation on GPUs. For images of size $512 \times 512$, our GPU implementation takes less than 1 second to produce state-of-the-art Poisson denoising performance.

High-Order Markov Random Fields and Their Applications in Cross-Language Speech Recognition

Abstract In this paper we study the cross-language speech emotion recognition using high-order Markov random fields, especially the application in Vietnamese speech emotion recognition. First, we extract the basic speech features including pitch frequency, formant frequency and short-term intensity. Based on the low level descriptor we further construct the statistic features including maximum, minimum, mean and standard deviation. Second, we adopt the high-order Markov random fields (MRF) to optimize the cross-language speech emotion model. The dimensional restrictions may be modeled by MRF. Third, based on the Vietnamese and Chinese database we analyze the efficiency of our emotion recognition system. We adopt the dimensional emotion model (arousal-valence) to verify the efficiency of MRF configuration method. The experimental results show that the high-order Markov random fields can improve the dimensional emotion recognition in the cross-language experiments, and the configuration method shows promising robustness over different languages.