Fully-connected Conditional Random Fields Research Articles

Semantic segmentation-based framework for concrete pouring progress monitoring by using multiple surveillance cameras

Traditional construction progress monitoring methods face challenges in real-time monitoring of concrete pouring due to performance limitations and issues with registration and occlusion. This study proposes a framework based on multi-camera semantic fusion for monitoring the progress of concrete pouring. The construction site images are first segmented into semantic probabilities by a deep neural network. Dempster-Shafer evidence theory is then applied for semantic fusion, providing a comprehensive depiction of the progress. To reduce registration errors with the Building Information Model (BIM), a real-time perspective transformation algorithm is proposed to compensate for slight camera movements. Finally, a semantic inference method based on fully-connected Conditional Random Fields (CRFs) is employed to address occlusion by leveraging semantic context and the BIM floor plan. Comparative analysis confirmed these modules’ performance, with a remarkable reduction in relative error from 9.60% to 0.26%, enabling great potential for the continuous real-time monitoring of concrete pouring progress.

Swin-T-NFC CRFs: An encoder–decoder neural model for high-precision UAV positioning via point cloud super resolution and image semantic segmentation

3D point cloud and remote sensing images have been the primary data types for the development of high-precision positioning systems. Equipped with a LiDAR and a camera, an Unmanned Aerial Vehicle (UAV) can explore an uncharted territory and gather both 3D scans and aerial images in real time to dynamically inspect the surroundings. However, the high cost of a high-resolution LiDAR hinders the development of the perception module of an UAV. Also, it is essential to adopt accurate image semantic segmentation (SemSeg) algorithms to better understand the sensing environment. As hardware advancement is ongoing, support from the software side is crucial. A promising strategy for cost control in building a LiDAR-based positioning system is through point cloud super-resolution (SupRes), a technique that improves the point cloud resolution via algorithms. This study investigates a deep learning-based framework that adopts a classic encoder–decoder structure for both point cloud SupRes and image SemSeg. Unlike prior studies that mainly use convolutional neural networks (CNNs) for feature extraction, our model, named Swin-T-NFC CRFs, consists of a Vision Transformer (ViT)-based encoder and a fully connected conditional random fields (FC-CRFs)-based decoder, connected via a pyramid pooling module and multiple skip connections. Moreover, both encoder and decoder are coupled with a shifted window strategy that allows cross-window connection. As such, patches from different windows of the feature map can participate in self-attention computation, leading to more powerful modeling ability. Experimental results demonstrate that our method can effectively boost the prediction accuracy, reduce the error, and consistently outperform the state-of-the-art methods on simulated/real-world point cloud datasets and the urban drone dataset version 6.

Improving the Performance of Convolutional Neural Network for the Segmentation of Optic Disc in Fundus Images Using Attention Gates and Conditional Random Fields

The localization and segmentation of optic disc (OD) in fundus images is a crucial step in the pipeline for detecting the early onset of retinal diseases, such as macular degeneration, diabetic retinopathy, glaucoma, etc. In this paper, we are proposing a novel convolutional neural network architecture for the precise segmentation of the OD in fundus images. We modify the basic architectures of DeepLab v3+ and U-Net models by integrating a novel attention module between the encoder and decoder to attain the finest accuracy. We also use fully-connected conditional random fields to further boost the performance of these architectures. We compare the results of our best proposed architecture against other established architectures for optic disc segmentation on our private dataset, as well as on publicly available datasets, namely, DRIONS-DB, RIM-ONE v.3, and DRISHTI-GS. The results obtained with the proposed method outperforms the existing methods in the literature.

Indoor scene understanding via RGB-D image segmentation employing depth-based CNN and CRFs

With the availability of low-cost depth-visual sensing devices, such as Microsoft Kinect, we are experiencing a growing interest in indoor environment understanding, at the core of which is semantic segmentation in RGB-D image. The latest research shows that the convolutional neural network (CNN) still dominates the image semantic segmentation field. However, down-sampling operated during the training process of CNNs leads to unclear segmentation boundaries and poor classification accuracy. To address this problem, in this paper, we propose a novel end-to-end deep architecture, termed FuseCRFNet, which seamlessly incorporates a fully-connected Conditional Random Fields (CRFs) model into a depth-based CNN framework. The proposed segmentation method uses the properties of pixel-to-pixel relationships to increase the accuracy of image semantic segmentation. More importantly, we formulate the CRF as one of the layers in FuseCRFNet to refine the coarse segmentation in the forward propagation, in meanwhile, it passes back the errors to facilitate the training. The performance of our FuseCRFNet is evaluated by experimenting with SUN RGB-D dataset, and the results show that the proposed algorithm is superior to existing semantic segmentation algorithms with an improvement in accuracy of at least 2%, further verifying the effectiveness of the algorithm.

Building Corner Detection in Aerial Images with Fully Convolutional Networks.

In aerial images, corner points can be detected to describe the structural information of buildings for city modeling, geo-localization, and so on. For this specific vision task, the existing generic corner detectors perform poorly, as they are incapable of distinguishing corner points on buildings from those on other objects such as trees and shadows. Recently, fully convolutional networks (FCNs) have been developed for semantic image segmentation that are able to recognize a designated kind of object through a training process with a manually labeled dataset. Motivated by this achievement, an FCN-based approach is proposed in the present work to detect building corners in aerial images. First, a DeepLab model comprised of improved FCNs and fully-connected conditional random fields (CRFs) is trained end-to-end for building region segmentation. The segmentation is then further improved by using a morphological opening operation to increase its accuracy. Corner points are finally detected on the contour curves of building regions by using a scale-space detector. Experimental results show that the proposed building corner detection approach achieves an F-measure of 0.83 in the test image set and outperforms a number of state-of-the-art corner detectors by a large margin.

Fully Connected Conditional Random Fields for High-Resolution Remote Sensing Land Use/Land Cover Classification with Convolutional Neural Networks

The interpretation of land use and land cover (LULC) is an important issue in the fields of high-resolution remote sensing (RS) image processing and land resource management. Fully training a new or existing convolutional neural network (CNN) architecture for LULC classification requires a large amount of remote sensing images. Thus, fine-tuning a pre-trained CNN for LULC detection is required. To improve the classification accuracy for high resolution remote sensing images, it is necessary to use another feature descriptor and to adopt a classifier for post-processing. A fully connected conditional random fields (FC-CRF), to use the fine-tuned CNN layers, spectral features, and fully connected pairwise potentials, is proposed for image classification of high-resolution remote sensing images. First, an existing CNN model is adopted, and the parameters of CNN are fine-tuned by training datasets. Then, the probabilities of image pixels belong to each class type are calculated. Second, we consider the spectral features and digital surface model (DSM) and combined with a support vector machine (SVM) classifier, the probabilities belong to each LULC class type are determined. Combined with the probabilities achieved by the fine-tuned CNN, new feature descriptors are built. Finally, FC-CRF are introduced to produce the classification results, whereas the unary potentials are achieved by the new feature descriptors and SVM classifier, and the pairwise potentials are achieved by the three-band RS imagery and DSM. Experimental results show that the proposed classification scheme achieves good performance when the total accuracy is about 85%.

Estimating Depth From Monocular Images as Classification Using Deep Fully Convolutional Residual Networks

Depth estimation from single monocular images is a key component of scene understanding and has benefited largely from deep convolutional neural networks (CNN) recently. In this article, we take advantage of the recent deep residual networks and propose a simple yet effective approach to this problem. We formulate depth estimation as a pixel-wise classification task. Specifically, we first discretize the continuous depth values into multiple bins and label the bins according to their depth range. Then we train fully convolutional deep residual networks to predict the depth label of each pixel. Performing discrete depth label classification instead of continuous depth value regression allows us to predict a confidence in the form of probability distribution. We further apply fully-connected conditional random fields (CRF) as a post processing step to enforce local smoothness interactions, which improves the results. We evaluate our approach on both indoor and outdoor datasets and achieve state-of-the-art performance.

Context-based abnormal object detection using the fully-connected conditional random fields

The contextual information plays an important role in computer vision, particularly in object detection and scene understanding. The existing contextual models use only the relationship between normal objects and natural scenes, and thus there still remains a difficult problem in detection of abnormal objects. This paper proposes an abnormal object detection model using the fully-connected conditional random fields to integrate the contextual information such as the co-occurrence and geometric relationships between objects. With this formulation, the proposed model combines the co-occurrence, spatial interaction between objects, and scale information. To this end, we use a feature embedding technique to find a geometry that reflects the statistical relationship in the pairwise term. Abnormal object detection is solved by using probabilistic variational inference such as the mean field approximation. Experimental results show that the proposed abnormal object detection model achieves significant improvement over the state-of-the-art models on the out-of-context dataset and abnormal object dataset.

BUILDING EXTRACTION FROM REMOTE SENSING DATA USING FULLY CONVOLUTIONAL NETWORKS

Abstract. Building detection and footprint extraction are highly demanded for many remote sensing applications. Though most previous works have shown promising results, the automatic extraction of building footprints still remains a nontrivial topic, especially in complex urban areas. Recently developed extensions of the CNN framework made it possible to perform dense pixel-wise classification of input images. Based on these abilities we propose a methodology, which automatically generates a full resolution binary building mask out of a Digital Surface Model (DSM) using a Fully Convolution Network (FCN) architecture. The advantage of using the depth information is that it provides geometrical silhouettes and allows a better separation of buildings from background as well as through its invariance to illumination and color variations. The proposed framework has mainly two steps. Firstly, the FCN is trained on a large set of patches consisting of normalized DSM (nDSM) as inputs and available ground truth building mask as target outputs. Secondly, the generated predictions from FCN are viewed as unary terms for a Fully connected Conditional Random Fields (FCRF), which enables us to create a final binary building mask. A series of experiments demonstrate that our methodology is able to extract accurate building footprints which are close to the buildings original shapes to a high degree. The quantitative and qualitative analysis show the significant improvements of the results in contrast to the multy-layer fully connected network from our previous work.