Fully-connected Conditional Random Field Research Articles

Open-Pit Mining Area Extraction from High-Resolution Remote Sensing Images Based on EMANet and FC-CRF

Rapid and accurate identification of open-pit mining areas is essential for guiding production planning and assessing environmental impact. Remote sensing technology provides an effective means for open-pit mine boundary identification. In this study, an effective method for delineating an open-pit mining area from remote sensing images is proposed, which is based on the deep learning model of the Expectation-Maximizing Attention Network (EMANet) and the fully connected conditional random field (FC-CRF) algorithm. First, ResNet-34 was applied as the backbone network to obtain preliminary features. Second, the EMA mechanism was used to enhance the learning of important information and details in the image. Finally, a postprocessing program based on FC-CRF was introduced to optimize the initial prediction results. Meanwhile, the extraction effect of MobileNetV3, U-Net, fully convolutional network (FCN), and our method were compared on the same data set for the open-pit mining areas. The advantage of the model is verified by the visual graph results, and the accuracy evaluation index based on the confusion matrix calculation. pixel accuracy (PA), mean intersection over union (MIoU), and kappa were 98.09%, 89.48%, and 88.48%, respectively. The evaluation results show that this method effectively identifies open-pit mining areas. It is of practical significance to complete the extraction task of open-pit mining areas accurately and comprehensively, which can be used for production management and environmental protection of open-pit mines.

Group Activity Recognition Based on Interaction Contextual Information in Videos Using Machine Learning

This paper is about recognizing multiple person actions occurring in videos, including individual actions, interactions,and group activities. In an environment, multiple people perform group actions such as walking in groupsand talking by facing each other. The model develops by retrieving individual person action from video sequencesby representing interactive contextual features among multiple people. The novelty of the proposed frameworkis the development of interactive action context descriptors (IAC) and classifying group activities using MachineLearning. Each individual person and other nearby people’s relative action score are encoded by IAC in thevideo frame. Individual person action descriptors are important clues for recognition of multiple person activityby developing interaction context. An action retrieval technique was formulated based on KNN for individualaction classification scores. This model also introduces Fully Connected Conditional Random Field (FCCRF) tolearn interaction context information among multiple people. FCCRF regularizes activity categorization by thespatial-temporal model. This paper also presents threshold processing to improve the performance of contextdescriptors. The experimental results compared to state-of-the-art approaches and demonstrated improvement inperformance for group activity recognition.

One-shot Video Graph Generation for Explainable Action Reasoning

Human action analysis is a critical yet challenging task for understanding diverse video content. Recently, to enable explainable reasoning of video actions, a spatio-temporal video graph structure was proposed to represent the video state changes at the semantic level. However, its requirement of tedious manual annotation of all the video frames is a serious limitation. Obviously, this approach would have a tremendously expanded applicability if the video graph generation process can be automated. In this paper, a One-Shot Video Graph (OSVG) generation approach is proposed for more effective explainable action reasoning, which only requires a one-time annotation of the objects in the starting frame of the video. We first estimate the predefined relevant objects across the temporal dimension by employing a proposed one-shot target-aware tracking strategy. This helps obtain the object locations and links objects simultaneously across all video frames. Then, the scene graph of each video frame can be constructed by an attribute detector and a relationship detector based on the estimated object locations. In addition, to further enhance the reasoning accuracy of performed actions, a video graph smoothing mechanism is designed with a fully-connected Conditional Random Field (CRF). By sequentially examining every state transition (including attributes and relationships) of the smoothed video graph, the actions occurring can be recognized using pre-defined rules. Experiments on the CAD-120++ dataset and a newly collected NTU RGBD++ dataset have verified that the proposed OSVG is able to outperform other state-of-the-art video action reasoning strategies on both state recognition and action recognition accuracy.

A 3D Grouped Convolutional Network Fused with Conditional Random Field and Its Application in Image Multi-target Fine Segmentation

Aiming at the utilization of adjacent image correlation information in multi-target segmentation of 3D image slices and the optimization of segmentation results, a 3D grouped fully convolutional network fused with conditional random fields (3D-GFCN) is proposed. The model takes fully convolutional network (FCN) as the image segmentation infrastructure, and fully connected conditional random field (FCCRF) as the post-processing tool. It expands the 2D convolution into 3D operations, and uses a shortcut-connection structure to achieve feature fusion of different levels and scales, to realizes the fine-segmentation of 3D image slices. 3D-GFCN uses 3D convolution kernel to correlate the information of 3D image adjacent slices, uses the context correlation and probability exploration mechanism of FCCRF to optimize the segmentation results, and uses the grouped convolution to reduce the model parameters. The dice loss that can ignore the influence of background pixels is used as the training objective function to reduce the influence of the imbalance quantity between background pixels and target pixels. The model can automatically study and focus on target structures of different shapes and sizes in the image, highlight the salient features useful for specific tasks. In the mechanism, it can improve the shortcomings and limitations of the existing image segmentation algorithms, such as insignificant morphological features of the target image, weak correlation of spatial information and discontinuous segmentation results, and improve the accuracy of multi-target segmentation results and learning efficiency. Take abdominal abnormal tissue detection and multi-target segmentation based on 3D computer tomography (CT) images as verification experiments. In the case of small-scale and unbalanced data set, the average Dice coefficient is 88.8%, the Class Pixel Accuracy is 95.3%, and Intersection of Union is 87.8%. Compared with other methods, the performance evaluation index and segmentation accuracy are significantly improved. It shows that the proposed method has good applicability for solving typical multi-target image segmentation problems, such as boundary overlap, offset deformation and low contrast.

Learning Pairwise Potential CRFs in Deep Siamese Network for Change Detection

Very high resolution (VHR) images change detection plays an important role in many remote sensing applications, such as military reconnaissance, urban planning and natural resource monitoring. Recently, fully connected conditional random field (FCCRF)-facilitated deep convolutional neural networks have shown promising results in change detection. However, the FCCRF in change detection currently is still postprocessing based on the output of the front-end network, which is not a convenient end-to-end network model and cannot combine front-end network knowledge with the knowledge of pairwise potential. Therefore, we propose a new end-to-end deep Siamese pairwise potential CRFs network (PPNet) for VHR images change detection. Specifically, this method adds a conditional random field recurrent neural network (CRF-RNN) unit into the convolutional neural network and integrates the knowledge of unary potential and pairwise potential in the end-to-end training process, aiming to refine the edges of changed areas and to remove the distant noise. In order to correct the front-end network identification errors, the method uses effective channel attention (ECA) to further effectively distinguish the change areas. Our experimental results on two data sets verify that the proposed method has more advanced capability with almost no increase in the number of parameters and effectively avoids the overfitting phenomenon in the training process.

A Semantic Segmentation Method for Remote Sensing Images Based on the Swin Transformer Fusion Gabor Filter

Semantic segmentation of remote sensing images is increasingly important in urban planning, autonomous driving, disaster monitoring, and land cover classification. With the development of high-resolution remote sensing satellite technology, multilevel, large-scale, and high-precision segmentation has become the focus of current research. High-resolution remote sensing images have high intraclass diversity and low interclass separability, which pose challenges to the precision of the detailed representation of multiscale information. In this paper, a semantic segmentation method for remote sensing images based on Swin Transformer fusion with a Gabor filter is proposed. First, a Swin Transformer is used as the backbone network to extract image information at different levels. Then, the texture and edge features of the input image are extracted with a Gabor filter, and the multilevel features are merged by introducing a feature aggregation module (FAM) and an attentional embedding module (AEM). Finally, the segmentation result is optimized with the fully connected conditional random field (FC-CRF). Our proposed method, called Swin-S-GF, its mean Intersection over Union (mIoU) scored 80.14%, 66.50%, and 70.61% on the large-scale classification set, the fine land-cover classification set, and the “AI + Remote Sensing imaging dataset” (AI+RS dataset), respectively. Compared with DeepLabV3, mIoU increased by 0.67%, 3.43%, and 3.80%, respectively. Therefore, we believe that this model provides a good tool for the semantic segmentation of high-precision remote sensing images.

Multi-class medical image segmentation using one-vs-rest graph cuts and majority voting.

Purpose: Semi-automatic image segmentation is still a valuable tool in clinical applications since it retains the expert oversights legally required. However, semi-automatic methods for simultaneous multi-class segmentation are difficult to be clinically implemented due to the complexity of underlining algorithms. We purpose an efficient one-vs-rest graph cut approach of which the complexity only grows linearly as the number of classes increases. Approach: Given an image slice, we construct multiple one-vs-rest graphs, each for a tissue class, for inference of a conditional random field (CRF). The one-vs-rest graph cut is to minimize the CRF energy derived from regional and boundary class probabilities estimated from random forests to obtain a one-vs-rest segmentation. The final segmentation is obtained by fusing from those one-vs-rest segmentations based on majority voting. We compare our method to a well-used multi-class graph cut method, alpha-beta swap, and a fully connected CRF (FCCRF) method, in brain tumor segmentation of 20 high-grade tumor cases in 2013 MICCAI dataset. Results: Our method achieved mean Dice score of 0.83 for whole tumor, compared to 0.80 by alpha-beta swap and 0.79 by FCCRF. There was a performance improvement over alpha-beta swap by a factor of five. Conclusions: Our method utilizes the probabilistic-based CRF which can be estimated from any machine learning technique. Comparing to traditional multi-class graph cut, the purposed one-vs-rest approach has complexity that grows only linearly as the number of classes increases, therefore, our method can be applicable for both online semi-automatic and offline automatic segmentation in clinical applications.

Weakly Supervised Deep Nuclei Segmentation Using Partial Points Annotation in Histopathology Images.

Nuclei segmentation is a fundamental task in histopathology image analysis. Typically, such segmentation tasks require significant effort to manually generate accurate pixel-wise annotations for fully supervised training. To alleviate such tedious and manual effort, in this paper we propose a novel weakly supervised segmentation framework based on partial points annotation, i.e., only a small portion of nuclei locations in each image are labeled. The framework consists of two learning stages. In the first stage, we design a semi-supervised strategy to learn a detection model from partially labeled nuclei locations. Specifically, an extended Gaussian mask is designed to train an initial model with partially labeled data. Then, self-training with background propagation is proposed to make use of the unlabeled regions to boost nuclei detection and suppress false positives. In the second stage, a segmentation model is trained from the detected nuclei locations in a weakly-supervised fashion. Two types of coarse labels with complementary information are derived from the detected points and are then utilized to train a deep neural network. The fully-connected conditional random field loss is utilized in training to further refine the model without introducing extra computational complexity during inference. The proposed method is extensively evaluated on two nuclei segmentation datasets. The experimental results demonstrate that our method can achieve competitive performance compared to the fully supervised counterpart and the state-of-the-art methods while requiring significantly less annotation effort.

Refined UNet: UNet-Based Refinement Network for Cloud and Shadow Precise Segmentation

Formulated as a pixel-level labeling task, data-driven neural segmentation models for cloud and corresponding shadow detection have achieved a promising accomplishment in remote sensing imagery processing. The limited capability of these methods to delineate the boundaries of clouds and shadows, however, is still referred to as a central issue of precise cloud and shadow detection. In this paper, we focus on the issue of rough cloud and shadow location and fine-grained boundary refinement of clouds on the dataset of Landsat8 OLI and therefore propose the Refined UNet to achieve this goal. To this end, a data-driven UNet-based coarse prediction and a fully-connected conditional random field (Dense CRF) are concatenated to achieve precise detection. Specifically, the UNet network with adaptive weights of balancing categories is trained from scratch, which can locate the clouds and cloud shadows roughly, while correspondingly the Dense CRF is employed to refine the cloud boundaries. Eventually, Refined UNet can give cloud and shadow proposals sharper and more precisely. The experiments and results illustrate that our model can propose sharper and more precise cloud and shadow segmentation proposals than the ground truths do. Additionally, evaluations on the Landsat 8 OLI imagery dataset of Blue, Green, Red, and NIR bands illustrate that our model can be applied to feasibly segment clouds and shadows on the four-band imagery data.

Oil palm plantation mapping from high-resolution remote sensing images using deep learning

ABSTRACT Oil palm plantation mapping is an important task in land planning and management in Malaysia. Most existing studies were based on satellite images using traditional machine learning or image segmentation methods. In order to obtain finer oil palm plantation maps from high spatial-resolution satellite images, we proposed a novel deep learning-based semantic segmentation approach, named Residual Channel Attention Network (RCANet). It consists of an encoder-decoder architecture and a post-processing component. The Residual Channel Attention Unit (RCAU) designed in our proposed approach reuses the low-level features extracted from the encoder part through upsampling, effectively enhancing the discriminative features and suppressing the indiscriminate features. We extended the fully connected Conditional Random Field (FC-CRF) in the post-processing to further refine the segmentation results. Experiment results were evaluated by our proposed Malaysian Oil Palm Plantation Dataset (MOPPD), which was collected from the Google Earth high spatial-resolution image and published in this article. Our proposed method achieves the overall accuracy (OA) of 96.88% and mean Intersection-over-Union (mean IoU) of 90.58%, improving the OA by 2.03%-3.96% and the mean IoU by 2.13%-5.44% compared with other semantic segmentation methods (i.e. Fully Connected Network, U-Net and Fully Connected DenseNet). In addition, we exhibited the results of the oil palm plantation mapping in large-scale areas (around 320 km2) and demonstrated the effectiveness of our method for large-scale mapping.

Deep Fully Connected Model for Collective Activity Recognition

Group activity recognition is a challenging task because there is an exponentially large number of semantic and geometrical relationships among individuals. This makes it difficult to model these interactions and merge them as a whole for group activity classification. In this paper, we propose a deep fully-connected model for group recognition, first we use the spatial-temporal model based on convolution neural network (CNN) and long short-term memory networks (LSTM) network to capture the dynamic features of each person. Then, we use the fully-connected conditional random field (FCCRF) to learn the interactions between people. Finally, with FCCRF potential functions we re-fine the activity recognition predicted by the spatial-temporal model. The experimental results on collective activity data-set and collective activity extended data-set show that our model improves recognition accuracy over baseline methods and gets competitive results in comparison to the state-of-the-art models.

Probabilistic Substrate Classification with Multispectral Acoustic Backscatter: A Comparison of Discriminative and Generative Models

We propose a probabilistic graphical model for discriminative substrate characterization, to support geological and biological habitat mapping in aquatic environments. The model, called a fully-connected conditional random field (CRF), is demonstrated using multispectral and monospectral acoustic backscatter from heterogeneous seafloors in Patricia Bay, British Columbia, and Bedford Basin, Nova Scotia. Unlike previously proposed discriminative algorithms, the CRF model considers both the relative backscatter magnitudes of different substrates and their relative proximities. The model therefore combines the statistical flexibility of a machine learning algorithm with an inherently spatial treatment of the substrate. The CRF model predicts substrates such that nearby locations with similar backscattering characteristics are likely to be in the same substrate class. The degree of allowable proximity and backscatter similarity are controlled by parameters that are learned from the data. CRF model results were evaluated against a popular generative model known as a Gaussian Mixture model (GMM) that doesn’t include spatial dependencies, only covariance between substrate backscattering response over different frequencies. Both models are used in conjunction with sparse bed observations/samples in a supervised classification. A detailed accuracy assessment, including a leave-one-out cross-validation analysis, was performed using both models. Using multispectral backscatter, the GMM model trained on 50% of the bed observations resulted in a 75% and 89% average accuracies in Patricia Bay and Bedford Basin, respectively. The same metrics for the CRF model were 78% and 95%. Further, the CRF model resulted in a 91% mean cross-validation accuracy across four substrate classes at Patricia Bay, and a 99.5% mean accuracy across three substrate classes at Bedford Basin, which suggest that the CRF model generalizes extremely well to new data. This analysis also showed that the CRF model was much less sensitive to the specific number and locations of bed observations than the generative model, owing to its ability to incorporate spatial autocorrelation in substrates. The CRF therefore may prove to be a powerful ‘spatially aware’ alternative to other discriminative classifiers.

An automatic change detection approach for rapid flood mapping in Sentinel-1 SAR data

In this paper, a two-step automatic change detection chain for rapid flood mapping based on Sentinel-1 Synthetic Aperture Radar (SAR) data is presented. First, a reference image is selected from a set of potential image candidates via a Jensen-Shannon (JS) divergence-based index. Second, saliency detection is applied on log-ratio data to derive the prior probabilities of changed and unchanged classes for initializing the following expectation-maximization (EM) based generalized Gaussian mixture model (GGMM). The saliency-guided GGMM is capable of capturing the primary pixel-based change information and handling highly imbalanced datasets. A fully-connected conditional random field (FCRF) model, which takes long-range pairwise potential connections into account, is integrated to remove the ambiguities of the saliency-guided GGMM and to achieve the final change map. The whole process chain is automatic with an efficient computation. The proposed approach was validated on flood events at the Evros River, Greece and the Wharfe River and Ouse River in York, United Kingdom. Kappa coefficients (k) of 0.9238 and 0.8682 were obtained respectively. The sensitivity analysis underlines the robustness of the proposed approach for rapid flood mapping.

WATERBODIES EXTRACTION FROM LANDSAT8-OLI IMAGERY USING AWATER INDEXS-GUIED STOCHASTIC FULLY-CONNECTED CONDITIONAL RANDOM FIELD MODEL AND THE SUPPORT VECTOR MACHINE

Abstract. One of the most important applications of remote sensing classification is water extraction. The water index (WI) based on Landsat images is one of the most common ways to distinguish water bodies from other land surface features. But conventional WI methods take into account spectral information only form a limited number of bands, and therefore the accuracy of those WI methods may be constrained in some areas which are covered with snow/ice, clouds, etc. An accurate and robust water extraction method is the key to the study at present. The support vector machine (SVM) using all bands spectral information can reduce for these classification error to some extent. Nevertheless, SVM which barely considers spatial information is relatively sensitive to noise in local regions. Conditional random field (CRF) which considers both spatial information and spectral information has proven to be able to compensate for these limitations. Hence, in this paper, we develop a systematic water extraction method by taking advantage of the complementarity between the SVM and a water index-guided stochastic fully-connected conditional random field (SVM-WIGSFCRF) to address the above issues. In addition, we comprehensively evaluate the reliability and accuracy of the proposed method using Landsat-8 operational land imager (OLI) images of one test site. We assess the method’s performance by calculating the following accuracy metrics: Omission Errors (OE) and Commission Errors (CE); Kappa coefficient (KP) and Total Error (TE). Experimental results show that the new method can improve target detection accuracy under complex and changeable environments.

RANDOM-FOREST-ENSEMBLE-BASED CLASSIFICATION OF HIGH-RESOLUTION REMOTE SENSING IMAGES AND NDSM OVER URBAN AREAS

Abstract. As an intermediate step between raw remote sensing data and digital urban maps, remote sensing data classification has been a challenging and long-standing research problem in the community of remote sensing. In this work, an effective classification method is proposed for classifying high-resolution remote sensing data over urban areas. Starting from high resolution multi-spectral images and 3D geometry data, our method proceeds in three main stages: feature extraction, classification, and classified result refinement. First, we extract color, vegetation index and texture features from the multi-spectral image and compute the height, elevation texture and differential morphological profile (DMP) features from the 3D geometry data. Then in the classification stage, multiple random forest (RF) classifiers are trained separately, then combined to form a RF ensemble to estimate each sample’s category probabilities. Finally the probabilities along with the feature importance indicator outputted by RF ensemble are used to construct a fully connected conditional random field (FCCRF) graph model, by which the classification results are refined through mean-field based statistical inference. Experiments on the ISPRS Semantic Labeling Contest dataset show that our proposed 3-stage method achieves 86.9% overall accuracy on the test data.

High-Resolution Remote Sensing Data Classification over Urban Areas Using Random Forest Ensemble and Fully Connected Conditional Random Field

As an intermediate step between raw remote sensing data and digital maps, remote sensing data classification has been a challenging and long-standing problem in the remote sensing research community. In this work, an automated and effective supervised classification framework is presented for classifying high-resolution remote sensing data. Specifically, the presented method proceeds in three main stages: feature extraction, classification, and classified result refinement. In the feature extraction stage, both multispectral images and 3D geometry data are used, which utilizes the complementary information from multisource data. In the classification stage, to tackle the problems associated with too many training samples and take full advantage of the information in the large-scale dataset, a random forest (RF) ensemble learning strategy is proposed by combining several RF classifiers together. Finally, an improved fully connected conditional random field (FCCRF) graph model is employed to derive the contextual information to refine the classification results. Experiments on the ISPRS Semantic Labeling Contest dataset show that the presented 3-stage method achieves 86.9% overall accuracy, which is a new state-of-the-art non-CNN (convolutional neural networks)-based classification method.

Noise-Compensated, Bias-Corrected Diffusion Weighted Endorectal Magnetic Resonance Imaging via a Stochastically Fully-Connected Joint Conditional Random Field Model.

Diffusion weighted magnetic resonance imaging (DW-MR) is a powerful tool in imaging-based prostate cancer screening and detection. Endorectal coils are commonly used in DW-MR imaging to improve the signal-to-noise ratio (SNR) of the acquisition, at the expense of significant intensity inhomogeneities (bias field) that worsens as we move away from the endorectal coil. The presence of bias field can have a significant negative impact on the accuracy of different image analysis tasks, as well as prostate tumor localization, thus leading to increased inter- and intra-observer variability. Retrospective bias correction approaches are introduced as a more efficient way of bias correction compared to the prospective methods such that they correct for both of the scanner and anatomy-related bias fields in MR imaging. Previously proposed retrospective bias field correction methods suffer from undesired noise amplification that can reduce the quality of bias-corrected DW-MR image. Here, we propose a unified data reconstruction approach that enables joint compensation of bias field as well as data noise in DW-MR imaging. The proposed noise-compensated, bias-corrected (NCBC) data reconstruction method takes advantage of a novel stochastically fully connected joint conditional random field (SFC-JCRF) model to mitigate the effects of data noise and bias field in the reconstructed MR data. The proposed NCBC reconstruction method was tested on synthetic DW-MR data, physical DW-phantom as well as real DW-MR data all acquired using endorectal MR coil. Both qualitative and quantitative analysis illustrated that the proposed NCBC method can achieve improved image quality when compared to other tested bias correction methods. As such, the proposed NCBC method may have potential as a useful retrospective approach for improving the consistency of image interpretations.

Could scene context be beneficial for scene text detection?

Scene text detection and scene segmentation are meaningful tasks in the computer vision field. Could the semantic scene segmentation assist scene text detection in any degree? For example, can we expect the probability of a region being text is low if its surrounding segment, i.e., its context, is labeled as sky? In this paper, we have a positive answer by constructing a scene context-based text detection model. In this model, we use texton features and a fully-connected conditional random field (CRF) to estimate pixel-level scene class׳s probability to be considered as image׳s context feature. Meanwhile, maximally stable extremal regions (MSERs) are extracted, integrated and extended as image patches of character candidates. Then, each image patch is fed to a simple two-layer convolutional neural network (CNN) to automatically extract its character feature. The averaged context feature of the corresponding patch is considered as the patch׳s context feature. The character feature and context feature are fused as the input into a support vector machine for text/non-text determination. Finally, as a post-processing, neighboring text regions are grouped hierarchically. The performance evaluation on ICDAR2013 and SVT databases, as well as a preliminary evaluation on a patch-level database, proves that the scene context can improve the performance of scene text detection. Moreover, the comparative study with state-of-the-art methods shows the top-level performance of our method.

A new change-detection method in high-resolution remote sensing images based on a conditional random field model

ABSTRACTA new change-detection method for remote sensing images based on a conditional random field (CRF) model is proposed in this paper. The method artfully uses memberships of Fuzzy C-means as unary potentials in the fully connected CRF (FCCRF) model without training parameters, and pairwise potentials of the CRF model are defined by a linear combination of Gaussian kernels, with which a highly efficient approximate inference algorithm can be used. The proposed FCCRF model is expressed on the complete set of pixels in both the observed multitemporal images, which can incorporate long range contextual information of remote-sensing images and enable greatly refined change-detection results. Experimental results demonstrate that the proposed approach leads to more accurate pixel-level change-detection performance and is more robust against noise than traditional algorithms.