Markov Random Field Optimization Research Articles

Weakly supervised land-cover classification of high-resolution images with low-resolution labels through optimized label refinement

ABSTRACT Land cover classification, or semantic segmentation, has been one of the most critical research areas in remote sensing (RS) and remains crucial for many downstream applications. Although deep learning (DL) based models have recently dominated this field, models trained using the dataset of one region generally cannot predict reliable classification results in other regions. Despite the abundance of high-resolution (HR) satellite images around the globe, not enough HR labels are available for building a one-for-all-purpose model. This study utilizes widely available low-resolution (LR) labels with weakly supervised methods to obtain land cover maps worldwide. Previous methods designed new learning components, such as loss functions, to fully use LR labels or trained DL models with LR labels to generate intermediate labels for further training and processing. Since this information is directly used or extracted from the original LR labels, methods without additional robustness are sensitive to noise in the labels, which causes error propagation in the results. On this basis, a novel label refinement approach is proposed that transforms noisy original LR labels into refined HR labels using two steps of noise filtering. First, based on spectral indices from the HR images, we select relatively confident labels from the LR labels through a Markov Random Field optimization framework. Second, a shallow classifier such as random forest (RF) is trained using the selected pixels to supplement previously unselected labels and refine low-confidence labels with new and high-confidence labels. The results showed that in the experiment on the Data Fusion Contest 2020 dataset, the semantic segmentation models trained using our refined HR labels had a 2–14% higher average accuracy than those trained using the original LR labels. They also outperformed other weakly supervised methods directly using original LR labels and had a 7.5% higher average accuracy than the winning method.

Improving the Accuracy of Brain Tumor Identification in Magnetic Resonanceaging using Super-pixel and Fast Primal Dual Algorithm

Brain tumors are one of the most common causes of death that have been widely investigated by scholars in research areas, including care and prevention. Despite various empirical studies on the brain tumor segmentatin, there is still a need for further investigation. This fact is more needed in the automatic methods of brain tumors detection. In the present study, a new method for improving brain tumor segmentation accuracy based on super-pixel and fast primal dual (PD) algorithms has been proposed. The proposed method detects brain tumor tissue in Flair-MRI imaging in BRATS2012 dataset. This method detects the primary borders of tumors using a super-pixel algorithm, and improves brain tumor borders using fast PD in Markov random field optimization. Then, post-processing processes are used to delete white brain areas. Finally, an active contour algorithm was employed to display tumor area. Different experiments were carried on the proposed method and qualitative and quantitative criteria such as dice similarity measure, accuracy and F-measure were used for evaluation. The obtained results showed the efficiency of the proposed method, such that in the accuracy and sensitivity of 86.59 and 88.57% and F1-Measure 86.37 were obtained, respectively.

Patch-Based Uncalibrated Photometric Stereo Under Natural Illumination.

This paper presents a photometric stereo method that works with unknown natural illumination without any calibration objects or initial guess of the target shape. To solve this challenging problem, we propose the use of an equivalent directional lighting model for small surface patches consisting of slowly varying normals, and solve each patch up to an arbitrary orthogonal ambiguity. We further build the patch connections by extracting consistent surface normal pairs via spatial overlaps among patches and intensity profiles. Guided by these connections, the local ambiguities are unified to a global orthogonal one through Markov Random Field optimization and rotation averaging. After applying the integrability constraint, our solution contains only a binary ambiguity, which could be easily removed. Experiments using both synthetic and real-world datasets show our method provides even comparable results to calibrated methods.

Sequential Clique Optimization for Unsupervised and Weakly Supervised Video Object Segmentation

A novel video object segmentation algorithm, which segments out multiple objects in a video sequence in unsupervised or weakly supervised manners, is proposed in this work. First, we match visually important object instances to construct salient object tracks through a video sequence without any user supervision. We formulate this matching process as the problem to find maximal weight cliques in a complete k-partite graph and develop the sequential clique optimization algorithm to determine the cliques efficiently. Then, we convert the resultant salient object tracks into object segmentation results and refine them based on Markov random field optimization. Second, we adapt the sequential clique optimization algorithm to perform weakly supervised video object segmentation. To this end, we develop a sparse-to-dense network to convert the point cliques into segmentation results. The experimental results demonstrate that the proposed algorithm provides comparable or better performances than recent state-of-the-art VOS algorithms.

3-D Instance Segmentation of MVS Buildings

We present a novel 3D instance segmentation framework for Multi-View Stereo (MVS) buildings in urban scenes. Unlike existing works focusing on semantic segmentation of urban scenes, the emphasis of this work lies in detecting and segmenting 3D building instances even if they are attached and embedded in a large and imprecise 3D surface model. Multi-view RGB images are first enhanced to RGBH images by adding a heightmap and are segmented to obtain all roof instances using a fine-tuned 2D instance segmentation neural network. Instance masks from different multi-view images are then clustered into global masks. Our mask clustering accounts for spatial occlusion and overlapping, which can eliminate segmentation ambiguities among multi-view images. Based on these global masks, 3D roof instances are segmented out by mask back-projections and extended to the entire building instances through a Markov random field optimization. A new dataset that contains instance-level annotation for both 3D urban scenes (roofs and buildings) and drone images (roofs) is provided. To the best of our knowledge, it is the first outdoor dataset dedicated to 3D instance segmentation with much more annotations of attached 3D buildings than existing datasets. Quantitative evaluations and ablation studies have shown the effectiveness of all major steps and the advantages of our multi-view framework over the orthophoto-based method.

IDENTIFICATION OF MISCLASSIFIED PIXELS IN SEMANTIC SEGMENTATION WITH UNCERTAINTY EVALUATION

Abstract. Classification, and in particular semantic segmentation, plays a major role in remote sensing. In remote sensing, the classes usually correspond to landcover or landuse types while the data elements are image pixels. The results are so-called semantically segmented pixels describing the content of the data for each pixel. The identification of misclassified pixels is essential to perceive the overall performance of the classification algorithm. In the case of semantic segmentation, it is typically done with ground truth labels. However, such ground truth labels are rare and mostly reserved for training only. Especially deep learning approaches are data-hungry algorithms requesting a lot of labeled examples. In this work, we explore the possibility of using Monte-Carlo dropout for the identification of model-induced misclassifications. In particular, we obtain uncertainty measures from several inferences induced by the Monte-Carlo dropout. Furthermore, we examine how Markov Random Field optimization can reduce the number of misclassifications and facilitate their identification. The extent to which uncertainties provide information about misclassifications is assessed. Our results allow detecting 51 % of the misclassifications using uncertainties. Application of Markov Random Field optimization leads to a reduction of the percentage of misclassifications while detecting 0.4 % more misclassifications as without.

Active Learning Based 3D Semantic Labeling From Images and Videos

3D semantic segmentation is one of the most fundamental problems for 3D scene understanding and has attracted much attention in the field of computer vision. In this paper, we propose an active learning based 3D semantic labeling method for large-scale 3D mesh model generated from images or videos. Taking as input a 3D mesh model reconstructed from the image based 3D modeling system, coupled with the calibrated images, our method outputs a fine 3D semantic mesh model in which each facet is assigned a semantic label. There are three major steps in our framework: 2D semantic segmentation, 2D-3D semantic fusion, and batch image selection. A limited annotation image set is first used to fine-tune a pre-trained semantic segmentation network for obtaining the pixel-wise semantic probability maps. Then all these maps are back-projected into 3D space and fused on the 3D mesh model using Markov Random Field optimization, thus yield a preliminary 3D semantic mesh model and a heat model showing each facet’s confidence. This 3D semantic model is used as a reliable supervisor to select the parts that are not well segmented for manual annotation to boost the performance of the 2D semantic segmentation network, as well as the 3D mesh labeling, in the next iteration. This Training-Fusion-Selection process continues until the label assignment of the 3D mesh model becomes steady. By this means, we significantly reduce the amount for annotation but not the labeling quality of 3D semantic models. Extensive experiments demonstrate the effectiveness and generalization ability of our method on a wide variety of datasets.

Partition-Merge: Distributed Inference and Modularity Optimization

This paper presents a novel meta-algorithm, Partition-Merge (PM), which takes existing centralized algorithms for graph computation and makes them distributed and faster. In a nutshell, PM divides the graph into small subgraphs using our novel randomized partitioning scheme, runs the centralized algorithm on each partition separately, and then stitches the resulting solutions to produce a global solution. We demonstrate the efficiency of the PM algorithm on two popular problems: computation of Maximum A Posteriori (MAP) assignment in an arbitrary pairwise Markov Random Field (MRF) and modularity optimization for community detection. We show that the resulting distributed algorithms for these problems become fast, which run in time linear in the number of nodes in the graph. Furthermore, PM leads to performance comparable – or even better – to that of the centralized algorithms as long as the graph has polynomial growth property. More precisely, if the centralized algorithm is a $\mathcal {C}-$ factor approximation with constant $\mathcal {C}\ge 1$ , the resulting distributed algorithm is a $(\mathcal {C}+\delta)$ -factor approximation for any small $\delta >0$ ; and even if the centralized algorithm is a non-constant (e.g., logarithmic) factor approximation, then the resulting distributed algorithm becomes a constant factor approximation. For general graphs, we compute explicit bounds on the loss of performance of the resulting distributed algorithm with respect to the centralized algorithm. To show the efficiency of our algorithm, we conducted extensive experiments both on real-world networks and on synthetic networks. The experiments demonstrate that the PM algorithm provides a good trade-off between accuracy and running time.

Semantic-Based Building Extraction from LiDAR Point Clouds Using Contexts and Optimization in Complex Environment.

The extraction of buildings has been an essential part of the field of LiDAR point clouds processing in recent years. However, it is still challenging to extract buildings from huge amount of point clouds due to the complicated and incomplete structures, occlusions and local similarities between different categories in a complex environment. Taking the urban and campus scene as examples, this paper presents a versatile and hierarchical semantic-based method for building extraction using LiDAR point clouds. The proposed method first performs a series of preprocessing operations, such as removing ground points, establishing super-points and using them as primitives for subsequent processing, and then semantically labels the raw LiDAR data. In the feature engineering process, considering the purpose of this article is to extract buildings, we tend to choose the features extracted from super-points that can describe building for the next classification. There are a portion of inaccurate labeling results due to incomplete or overly complex scenes, a Markov Random Field (MRF) optimization model is constructed for postprocessing and segmentation results refinement. Finally, the buildings are extracted from the labeled points. Experimental verification was performed on three datasets in different scenes, our results were compared with the state-of-the-art methods. These evaluation results demonstrate the feasibility and effectiveness of the proposed method for extracting buildings from LiDAR point clouds in multiple environments.

Non-Rigid Multi-Modal 3D Medical Image Registration Based on Foveated Modality Independent Neighborhood Descriptor.

The non-rigid multi-modal three-dimensional (3D) medical image registration is highly challenging due to the difficulty in the construction of similarity measure and the solution of non-rigid transformation parameters. A novel structural representation based registration method is proposed to address these problems. Firstly, an improved modality independent neighborhood descriptor (MIND) that is based on the foveated nonlocal self-similarity is designed for the effective structural representations of 3D medical images to transform multi-modal image registration into mono-modal one. The sum of absolute differences between structural representations is computed as the similarity measure. Subsequently, the foveated MIND based spatial constraint is introduced into the Markov random field (MRF) optimization to reduce the number of transformation parameters and restrict the calculation of the energy function in the image region involving non-rigid deformation. Finally, the accurate and efficient 3D medical image registration is realized by minimizing the similarity measure based MRF energy function. Extensive experiments on 3D positron emission tomography (PET), computed tomography (CT), T1, T2, and (proton density) PD weighted magnetic resonance (MR) images with synthetic deformation demonstrate that the proposed method has higher computational efficiency and registration accuracy in terms of target registration error (TRE) than the registration methods that are based on the hybrid L-BFGS-B and cat swarm optimization (HLCSO), the sum of squared differences on entropy images, the MIND, and the self-similarity context (SSC) descriptor, except that it provides slightly bigger TRE than the HLCSO for CT-PET image registration. Experiments on real MR and ultrasound images with unknown deformation have also be done to demonstrate the practicality and superiority of the proposed method.

Multi-frame stereo matching with edges, planes, and superpixels

Abstract We present a multi-frame narrow-baseline stereo matching algorithm based on extracting and matching edges across multiple frames. Edge matching allows us to focus on the important features at the very beginning, and deal with occlusion boundaries as well as untextured regions. Given the initial sparse matches, we fit overlapping local planes to form a coarse, over-complete representation of the scene. After breaking up the reference image in our sequence into superpixels, we perform a Markov random field optimization to assign each superpixel to one of the plane hypotheses. Finally, we refine our continuous depth map estimate using a piecewise-continuous variational optimization. Our approach successfully deals with depth discontinuities, occlusions, and large textureless regions, while also producing detailed and accurate depth maps. We show that our method out-performs competing methods on high-resolution multi-frame stereo benchmarks and is well-suited for view interpolation applications.

Inpainting of Wide-Baseline Multiple Viewpoint Video.

We describe a non-parametric algorithm for multiple-viewpoint video inpainting. Uniquely, our algorithm addresses the domain of wide baseline multiple-viewpoint video (MVV) with no temporal look-ahead in near real time speed. A Dictionary of Patches (DoP) is built using multi-resolution texture patches reprojected from geometric proxies available in the alternate views. We dynamically update the DoP over time, and a Markov Random Field optimisation over depth and appearance is used to resolve and align a selection of multiple candidates for a given patch, this ensures the inpainting of large regions in a plausible manner conserving both spatial and temporal coherence. We demonstrate the removal of large objects (e.g., people) on challenging indoor and outdoor MVV exhibiting cluttered, dynamic backgrounds and moving cameras.

Multi-hypothesis contextual modeling for semantic segmentation

Semantic segmentation (i.e. image parsing) aims to annotate each image pixel with its corresponding semantic class label. Spatially consistent labeling of the image requires an accurate description and modeling of the local contextual information. Segmentation result is typically improved by Markov Random Field (MRF) optimization on the initial labels. However this improvement is limited by the accuracy of initial result and how the contextual neighborhood is defined. In this paper, we develop generalized and flexible contextual models for segmentation neighborhoods in order to improve parsing accuracy. Instead of using a fixed segmentation and neighborhood definition, we explore various contextual models for fusion of complementary information available in alternative segmentations of the same image. In other words, we propose a novel MRF framework that describes and optimizes the contextual dependencies between multiple segmentations. Simulation results on two common datasets demonstrate significant improvement in parsing accuracy over the baseline approaches.

$M^{4}CD$ : A Robust Change Detection Method for Intelligent Visual Surveillance

In this paper, we propose a robust change detection method for intelligent visual surveillance. This method, named $M^{4}CD$ , includes three major steps. First, a sample-based background model that integrates color and texture cues is built and updated over time. Second, multiple heterogeneous features (including brightness variation, chromaticity variation, and texture variation) are extracted by comparing the input frame with the background model, and a multi-view learning strategy is designed to online estimate the probability distributions for both foreground and background. The three features are approximately conditionally independent, making multi-view learning feasible. Pixel-wise foreground posteriors are then estimated with Bayes rule. Finally, the Markov random field (MRF) optimization and heuristic post-processing techniques are used sequentially to improve accuracy. In particular, a two-layer MRF model is constructed to represent pixel-based and superpixel-based contextual constraints compactly. Experimental results on the CDnet dataset indicate that $M^{4}CD$ is robust under complex environments and ranks among the top methods.

Depth Estimation for Lytro Images by Adaptive Window Matching on EPI

A depth estimation algorithm from plenoptic images is presented. There are two stages to estimate the depth. First is the initial estimation base on the epipolar plane images (EPIs). Second is the refinement of the estimations. At the initial estimation, adaptive window matching is used to improve the robustness. The size of the matching window is based on the texture description of the sample patch. Based on the texture entropy, a smaller window is used for a fine texture. A smooth texture requires a larger window. With the adaptive window size, different reference patches based on various depth are constructed. Then the depth estimation compares the similarity among those patches to find the best matching patch. To improve the initial estimation, a refinement algorithm based on the Markov Random Field (MRF) optimization is used. An energy function keeps the data similar to the original estimation, and then the data are smoothed by minimizing the second derivative. Depth values should satisfy consistency across multiple views.

A new image registration algorithm using SDTR

Accurate image registration is a vital step in many computer vision processes. However, traditional SIFT based methods are not able to obtain satisfactory results in some cases. In this paper, we turn the matching problem into a Markov Random Field (MRF) optimization problem and propose a fast and accurate image registration method based on message passing to screen out mismatches caused by the conventional SIFT-based algorithm. The proposed method comprises three steps. First, a SIFT detector is used to detect key points and then these points are described by Daisy instead of the SIFT detector (to improve efficiency); second, a new geometric constraint with distance and direction terms is designed and incorporated into sequential tree-reweighted message passing (TRW-S) this obtains the coarse matching result; finally, the RANSAC algorithm is adopted to filter out the mismatches and obtain an accurate transformation. The experimental results demonstrate that the proposed method significantly improves the efficiency and the accuracy of the registration process and secures a desirable result.

Subpixel Mapping of Multispectral Images Using Markov Random Field With Graph Cut Optimization

This letter presents a novel subpixel mapping method based on Markov random field (MRF) and graph cut optimization. First, the support vector regression is applied to generate the fractional images of classes based on randomly selected training pixels. Second, the spatially adaptive MRF-based subpixel mapping method is adopted to formulate the energy function. Third, to minimize the energy function, graph cut is applied. Finally, the subpixel vector border is extracted and integrated as a weighting function to fine-tune the smoothing parameters. We conduct the experiments on a downsampled SPOT-7 multispectral image and two synthetic images with regular and irregular objects, respectively. The results prove that graph cut can be an alternative method to the commonly used simulated annealing for the energy function optimization in MRF-based subpixel mapping, considering its much higher efficiency and comparable classification accuracy. Moreover, the integration of subpixel edge information is helpful to improve the accuracy of subpixel mapping results.

A Weighted Mirror Descent Algorithm for Nonsmooth Convex Optimization Problem

Large-scale nonsmooth convex optimization is a common problem for a range of computational areas including machine learning and computer vision. Problems in these areas contain special domain structures and characteristics. Special treatment of such problem domains, exploiting their structures, can significantly reduce the computational burden. In this paper, we consider a Mirror Descent method with a special choice of distance function for solving nonsmooth optimization problems over a Cartesian product of convex sets. We propose to use a nonlinear weighted distance in the projection step. The convergence analysis identifies optimal weighting parameters that, eventually, lead to the optimally weighted step-size strategy for every projection on a corresponding convex set. We show that the optimality bound of the Mirror Descent algorithm using the weighted distance is either an improvement to, or in the worst case as good as, the optimality bound of the Mirror Descent using unweighted distances. We demonstrate the efficiency of the algorithm by solving the Markov Random Fields optimization problem. In order to exploit the domain of the problem, we use a weighted log-entropy distance and a weighted Euclidean distance. Promising experimental results demonstrate the effectiveness of the proposed method.

Fast background subtraction for moving cameras based on nonparametric models

In this paper, a fast background subtraction algorithm for freely moving cameras is presented. A nonparametric sample consensus model is employed as the appearance background model. The as-similar-as-possible warping technique, which obtains multiple homographies for different regions of the frame, is introduced to robustly estimate and compensate the camera motion between the consecutive frames. Unlike previous methods, our algorithm does not need any preprocess step for computing the dense optical flow or point trajectories. Instead, a superpixel-based seeded region growing scheme is proposed to extend the motion cue based on the sparse optical flow to the entire image. Then, a superpixel-based temporal coherent Markov random field optimization framework is built on the raw segmentations from the background model and the motion cue, and the final background/foreground labels are obtained using the graph-cut algorithm. Extensive experimental evaluations show that our algorithm achieves satisfactory accuracy, while being much faster than the state-of-the-art competing methods.

A Novel MRF-Based Multifeature Fusion for Classification of Remote Sensing Images

The spatial information has been proved to be effective in improving the performance of spectral-based classification. However, it is difficult to describe different image scenes by using monofeature owing to complexity of the geospatial scenes. In this letter, a novel framework is developed to combine the multiple spectral and spatial features based on the Markov random field (MRF). Specifically, the pixels in an image are separated into reliable and unreliable ones according to the decision of multifeature classifications. The labels of the reliable pixels can be conveniently determined, but the unreliable pixels are then classified by fusing the multifeature classification results and reducing the classification uncertainties based on the MRF optimization. Experiments are conducted on three multispectral high-resolution images to verify the effectiveness of the proposed method. Several state-of-the-art multifeature classification methods are also achieved for the purpose of comparison. Moreover, three classifiers (i.e., multinomial logistic regression, support vector machines, and random forest) are used to test the performance of the proposed framework. It is shown that the proposed method can effectively integrate multiple features, yield promising results, and outperform other approaches compared.