Continuous Conditional Random Fields Research Articles

Short-Term Wind Power Forecast Based on Continuous Conditional Random Field

Short-Term Wind Power Forecast Based on Continuous Conditional Random Field

Visualization of Football Tactics with Deep Learning Models

In the last several years, computer vision tasks involving visual identification and tracking have seen a rise in the usage of deep learning technologies in recent years. An extremely difficult but rewarding endeavor is identifying and following football players’ targets. This may be used to study football tactical visualization. Due to the similar appearance and frequent occlusion of targets in football video, traditional methods often can only segment targets such as players and balls in the image but cannot track them or can only track them for a short time. Based on the related research of computer vision and deep learning, using several cameras, this study develops a system that can properly monitor many targets in a football stadium for a lengthy period of time. The main research contents of this paper are as follows: (1) a CNN for target displacement prediction is proposed, which no longer relies on the previous linear motion model or quadratic motion model, so that the multitarget tracking algorithm can be applied to more scenes. (2) For the first time in a multitarget tracking algorithm, a continuous conditional random field is used to model the asymmetric nature of the target relationship. At the same time, the CNN for target displacement prediction can be cascaded with the continuous conditional random field for end-to-end training, which greatly reduces the training difficulty. The parameters of the experiment in this paper are simple, and comprehensive and systematic experiments verify the validity and correctness of this work from different aspects.

Ice Thickness From Deep Learning and Conditional Random Fields: Application to Ice-Penetrating Radar Data With Radiometric Validation

Identifying the location of the ice-bedrock interface of glaciers and ice sheets is crucial for a wide range of geophysical applications, such as searching for liquid water in basal regions and computing ice thickness to quantify ice sheet and glacier mass balance. Simple, record-by-record, approaches to detecting the bottom of the ice echo may be affected by spurious off-nadir noise that requires significant manual interaction to correct. In this paper, we propose a deep learning model based on convolutional neural networks (CNNs) and continuous conditional random fields (CCRFs) to automate ice bed identification and better capture fine-grained basal detail. We deploy this approach on HiCARS radargrams, the first time deep learning methods have been applied to this dataset. Intuitively, our CNN captures the global geometry of the ice bed, while the CCRF adjusts initial CNN outputs to better incorporate fine-scale spatial information into the final prediction. We also develop a coherent geophysical framework using three echo characters (along-track continuity, relative delay, and signal coherency) to compare our model’s outputs with those of a manually targeted approach. Our analysis suggests that our CNN + CCRF model is as suitable as the manual approach for radiometric applications, and it outperforms the manual technique in identifying the first continuous return, which is most often the near-nadir reflection. Thus, our approach is more universal than the current manual labeling methodology in the range of geophysical applications where it can be used, and it provides better confidence regarding the source of the basal return detected.

Deep Autoencoder for Hyperspectral Unmixing via Global-Local Smoothing

Hyperspectral unmixing is to decompose the mixed pixels into pure spectral signatures (endmembers) and their proportions (abundances). Recently, deep learning-based methods have been applied to enhance the representation ability of unmixing models by extracting joint spatial–spectral characteristics of the hyperspectral data. However, most deep learning based-unmixing methods usually conduct global smoothing by convolutions on the whole hyperspectral imagery, which may ignore the variations within the imagery and result in oversmoothing. In this article, we propose a deep network for hyperspectral unmixing based on a new global–local smoothing autoencoder (GLA). GLA is an unsupervised model, which aims at exploring the local homogeneity and the global self-similarity of hyperspectral imagery. The proposed GLA network mainly includes two modules: a Local Continuous conditional random field Smoothing (LCS) module and a global recurrent smoothing (GRS) module. In LCS, we propose a conditional random field-based smoothing strategy to describe the joint spatial–spectral information within a local homogeneity region, which also reduces the risk of abundance maps boundary blurry. In GRS, we follow the self-similarity assumption for hyperspectral imagery and develop a recurrent neural network structure to exploit potential long-distance dependency relationships among pixels. The GLA is compared with several state-of-the-art unmixing methods on both real and synthetic data, and the abundance estimation results indicate that our method is promising. We will publish the code of GLA if this article has the honor to be accepted.

Continuous conditional random field convolution for point cloud segmentation

Point cloud segmentation is the foundation of 3D environmental perception for modern intelligent systems. To solve this problem and image segmentation, conditional random fields (CRFs) are usually formulated as discrete models in label space to encourage label consistency, which is actually a kind of postprocessing. In this paper, we reconsider the CRF in feature space for point cloud segmentation because it can capture the structure of features well to improve the representation ability of features rather than simply smoothing. Therefore, we first model the point cloud features with a continuous quadratic energy model and formulate its solution process as a message-passing graph convolution, by which it can be easily integrated into a deep network. We theoretically demonstrate that the message passing in the graph convolution is equivalent to the mean-field approximation of a continuous CRF model. Furthermore, we build an encoder-decoder network based on the proposed continuous CRF graph convolution (CRFConv), in which the CRFConv embedded in the decoding layers can restore the details of high-level features that were lost in the encoding stage to enhance the location ability of the network, thereby benefiting segmentation. Analogous to the CRFConv, we show that the classical discrete CRF can also work collaboratively with the proposed network via another graph convolution to further improve the segmentation results. Experiments on various point cloud benchmarks demonstrate the effectiveness and robustness of the proposed method. Compared with the state-of-the-art methods, the proposed method can also achieve competitive segmentation performance.

Optimization of IoT-Based Artificial Intelligence Assisted Telemedicine Health Analysis System

This paper presents an in-depth study and exploration of the health IoT architecture and related implementation technologies from both theoretical and practical aspects, with important theoretical significance and practical application value. The research includes cloud fusion health IoT architecture, multimodal information acquisition in health IoT perception layer, multi-level service quality assurance of health IoT based on human LAN, and emotional perception and emotional interaction in health IoT. In terms of health IoT architecture, the cloud convergence health IoT architecture is proposed to deeply integrate the health cloud platform and perception layer by integrating multiple communication technologies to optimize the user experience and make health IoT applications more closely connected with people. This paper describes the basic concepts and main components of multimodal sensing information collection, the design and implementation of a health monitoring cloud robotics platform, robotics-based multimodal data sensing and aggregation, and high comfort sustainable physiological signal collection based on smart clothes. The feasibility and performance of the QoS framework proposed in this paper are verified by computer simulations. In this paper, migration learning is used to implement emotion data labeling, continuous conditional random fields to identify emotions based on data collected from smartphones and smart clothes, respectively, and finally decision layer fusion for emotion classification prediction.

Distance-To-Mean Continuous Conditional Random Fields: Case Study in Traffic Congestion

Traffic prediction techniques are classified as having parametric, non-parametric, and a combination of parametric and non-parametric characteristics. The extreme learning machine (ELM) is a non-parametric technique that is commonly used to enhance traffic prediction problems. In this study, a modified probability approach, continuous conditional random fields (CCRF), is proposed and implemented with the ELM and then utilized to assess highway traffic data. The modification is conducted to improve the performance of non-parametric techniques, in this case, the ELM method. This proposed method is then called the distance-to-mean continuous conditional random fields (DM-CCRF). The experimental results show that the proposed technique suppresses the prediction error of the prediction model compared to the standard CCRF. The comparison between ELM as a baseline regressor, the standard CCRF, and the modified CCRF is displayed. The performance evaluation of the techniques is obtained by analyzing their mean absolute percentage error (MAPE) values. DM-CCRF is able to suppress the prediction model error to ~ 17.047 % , which is twice as good as that of the standard CCRF method. Based on the attributes of the dataset, the DM-CCRF method is better for the prediction of highway traffic than the standard CCRF method and the baseline regressor.

Vigilance Estimation Using a Wearable EOG Device in Real Driving Environment

Vigilance decrement in driving tasks has been reported to be a major factor in fatal accidents and could severely endanger public transportation safety. However, efficient approaches for estimating vigilance in real driving environment are still lacking. In this paper, we propose a novel approach for implementing continuous vigilance estimation using forehead electrooculograms (EOGs) acquired by wearable dry electrodes in both simulated and real driving environments. To improve the feasibility of this approach for real-world applications, a forehead EOG-based electrode placement with only four electrodes is designed. Flexible dry electrodes and an acquisition board are integrated as a wearable device for recording EOGs. Twenty and ten subjects participated in the simulated and real-world driving environment experiments, respectively. Accurate eye movement parameters from eye-tracking glasses are extracted to calculate the PERCLOS index for vigilance annotation. This is because the vigilance state is a temporally dynamic process, and a continuous conditional random field and a continuous conditional neural field are introduced to construct more accurate vigilance estimation models. To evaluate the efficiency of our system, systematic experiments are performed in real scenarios under various illumination and weather conditions following laboratory simulations as preliminary studies. The experimental results demonstrate that the wearable dry electrode prototype, which has a relatively comfortable forehead setup, can efficiently capture vigilance dynamics. The best mean correlation coefficients achieved by our proposed approach are 71.18% and 66.20% in laboratory simulations and real-world driving environments, respectively. The cross-environment experiments are performed to evaluate the simulated-to-real generalization and a best mean correlation coefficient of 53.96% is achieved.

Superpixel based continuous conditional random field neural network for semantic segmentation

Recently, unifying conditional random fields (CRFs) into convolutional neural network (CNN) and enlarging spatial feature maps are two principal strategies for improving semantic segmentation performance. It is observed that building CRF defined on full resolution features into CNN can generate segmentation masks of better quality, however there exist several challenges related to training efficiency and computation costs. To design a more powerful segmentation model by using CRF and full resolution features, this paper proposes a novel fully supervised scheme for semantic segmentation based on superpixels and continuous CRF (C-CRF). Our new architecture includes three subnetworks: a unary network, a pairwise network and a superpixel based continuous CRF network (SCCN). The unary network upsamples low resolution features via de-convolutional structures, while pairwise network learns pixel-level pairwise similarities. The outputs of the two networks are fed to the SCCN that consists of two differentiable superpixel pooling layers and a differentiable C-CRF layer, leading to inference between superpixels in an image. The whole framework is trained in an end-to-end way by the jointly pixel-level and superpixel-level supervised learning strategy. Besides, C-CRF inference is fused with pixel-level prediction during the test procedure to decrease the effect of superpixel quality. In the experiments, we evaluate the power of the three subnetworks and the learning strategy comprehensively. Results on three benchmark datasets demonstrate that compared with baseline networks and other CRF based semantic segmentation methods, performance gain can be achieved by using the proposed SCCN.

Radar and Rain Gauge Merging-Based Precipitation Estimation via Geographical–Temporal Attention Continuous Conditional Random Field

An accurate, high-resolution precipitation estimation based on rain gauge and radar observations is essential in various meteorological applications. Although numerous studies have demonstrated the effectiveness of merging two information sources rather than using separate sources, approaches that simultaneously consider the local radar reflectivity, the neighborhood rain gauge observations, and the temporal information are much less common. In this paper, we present a new framework for real-time quantitative precipitation estimation (QPE). By formulating the QPE as a continuous conditional random field (CCRF) learning problem, the spatiotemporal correlations of precipitation can be explored more thoroughly. Based on the CCRF, we further improve the accuracy of the precipitation estimation by introducing geographical and temporal attention. Specifically, we first present a data-driven weighting scheme to merge the first law of geography into the proposed framework, and hence, the neighborhood sample closer to the estimated grid can receive more attention. Second, the temporal attention penalizes the similarity between two adjacent timestamps via the discrepancy of two-view estimates, which can model the local temporal consistency and tolerate some drastic changes. A sufficient evaluation is conducted on 11 rainfall processes that occurred in 2015, and the results confirm the advantage of our proposal for real-time precipitation estimation.

FC-RCCN: Fully convolutional residual continuous CRF network for semantic segmentation

Enlarging the spatial resolution of features generated by fully convolutional networks (FCNs) can improve the performance of semantic segmentation. To achieve this goal, deeper network with deconvolutional structure can be applied. However, when the network architecture becomes more complex, the training efficiency may degrade. To address the joint optimization problem of improving spatial resolution through deeper networks and training deeper networks more effectively, we propose a Fully Convolutional Residual Continuous CRF Network (FC-RCCN) for semantic segmentation. FC-RCCN is composed of three subnetworks: a unary network, a pairwise network, and a superpixel based continuous conditional random filed (C-CRF) network. In order to generate full spatial resolution predictions with high-quality, a residual block based unary network with multi-scale features fusion is proposed. Even though the unary network is a deeper network, the whole framework can be trained effectively in an end-to-end way using the joint pixel-level and superpixel-level supervised learning strategy which is optimized by a pixel-level softmax cross entropy loss and a superpixel-level log-likelihood loss. Besides, C-CRF inference is fused with pixel-level prediction during the test procedure, which guarantees the method’s robustness to the superpxiel errors. In the experiments, we evaluatee the power of the three subnetworks and the learning strategy comprehensively. Experiments on three benchmark datasets demonstrate that the proposed FC-RCCN outperforms previous segmentation methods and obtains the state-of-the-art performance.

Learning Customer Behaviors for Effective Load Forecasting

Load forecasting has been deeply studied because of its critical role in Smart Grid. In current Smart Grid, there are various types of customers with different energy consumption patterns. Customer’s energy consumption patterns are referred to as customer behaviors. It would significantly benefit load forecasting in a grid if customer behaviors could be taken into account. This paper proposes an innovative method that aggregates different types of customers by their identified behaviors, and then predicts the load of each customer cluster, so as to improve load forecasting accuracy of the whole grid. Sparse Continuous Conditional Random Fields (sCCRF) is proposed to effectively identify different customer behaviors through learning. A hierarchical clustering process is then introduced to aggregate customers according to the identified behaviors. Within each customer cluster, a representative sCCRF is fine-tuned to predict the load of its cluster. The final load of the whole grid is obtained by summing the loads of each cluster. The proposed method for load forecasting in Smart Grid has two major advantages. 1) Learning customer behaviors not only improves the prediction accuracy but also has a low computational cost. 2) sCCRF can effectively model the load forecasting problem of one customer, and simultaneously select key features to identify its energy consumption pattern. Experiments conducted from different perspectives demonstrate the advantages of the proposed load forecasting method. Further discussion is provided, indicating that the approach of learning customer behaviors can be extended as a general framework to facilitate decision making in other market domains.

Deep Continuous Conditional Random Fields With Asymmetric Inter-Object Constraints for Online Multi-Object Tracking

Online Multi-Object Tracking (MOT) is a challenging problem and has many important applications including intelligence surveillance, robot navigation and autonomous driving. In existing MOT methods, individual object's movements and inter-object relations are mostly modeled separately and relations between them are still manually tuned. In addition, inter-object relations are mostly modeled in a symmetric way, which we argue is not an optimal setting. To tackle those difficulties, in this paper, we propose a Deep Continuous Conditional Random Field (DCCRF) for solving the online MOT problem in a track-by-detection framework. The DCCRF consists of unary and pairwise terms. The unary terms estimate tracked objects' displacements across time based on visual appearance information. They are modeled as deep Convolution Neural Networks, which are able to learn discriminative visual features for tracklet association. The asymmetric pairwise terms model inter-object relations in an asymmetric way, which encourages high-confidence tracklets to help correct errors of low-confidence tracklets and not to be affected by low-confidence ones much. The DCCRF is trained in an end-to-end manner for better adapting the influences of visual information as well as inter-object relations. Extensive experimental comparisons with state-of-the-arts as well as detailed component analysis of our proposed DCCRF on two public benchmarks demonstrate the effectiveness of our proposed MOT framework.

Monocular Depth Estimation Using Multi-Scale Continuous CRFs as Sequential Deep Networks.

Depth cues have been proved very useful in various computer vision and robotic tasks. This paper addresses the problem of monocular depth estimation from a single still image. Inspired by the effectiveness of recent works on multi-scale convolutional neural networks (CNN), we propose a deep model which fuses complementary information derived from multiple CNN side outputs. Different from previous methods using concatenation or weighted average schemes, the integration is obtained by means of continuous Conditional Random Fields (CRFs). In particular, we propose two different variations, one based on a cascade of multiple CRFs, the other on a unified graphical model. By designing a novel CNN implementation of mean-field updates for continuous CRFs, we show that both proposed models can be regarded as sequential deep networks and that training can be performed end-to-end. Through an extensive experimental evaluation, we demonstrate the effectiveness of the proposed approach and establish new state of the art results for the monocular depth estimation task on three publicly available datasets, i.e., NYUD-V2, Make3D and KITTI.

Depth Reconstruction from Single Images Using a Convolutional Neural Network and a Condition Random Field Model.

This paper presents an effective approach for depth reconstruction from a single image through the incorporation of semantic information and local details from the image. A unified framework for depth acquisition is constructed by joining a deep Convolutional Neural Network (CNN) and a continuous pairwise Conditional Random Field (CRF) model. Semantic information and relative depth trends of local regions inside the image are integrated into the framework. A deep CNN network is firstly used to automatically learn a hierarchical feature representation of the image. To get more local details in the image, the relative depth trends of local regions are incorporated into the network. Combined with semantic information of the image, a continuous pairwise CRF is then established and is used as the loss function of the unified model. Experiments on real scenes demonstrate that the proposed approach is effective and that the approach obtains satisfactory results.

Saliency Detection by Fully Learning a Continuous Conditional Random Field

Salient object detection is aimed at detecting and segmenting objects that human eyes are most focused on when viewing a scene. Recently, conditional random field (CRF) is drawn renewed interest, and is exploited in this field. However, when utilizing a CRF with unary and pairwise potentials having essential parameters, most existing methods only employ manually designed parameters, or learn parameters partly for the unary potentials. Observing that the saliency estimation is a continuous labeling issue, this paper proposes a novel data-driven scheme based on a special CRF framework, the so-called continuous CRF (C-CRF), where parameters for both unary and pairwise potentials are jointly learned. The proposed C-CRF learning provides an optimal way to integrate various unary saliency features with pairwise cues to discover salient objects. To the best of our knowledge, the proposed scheme is the first to completely learn a C-CRF for saliency detection. In addition, we propose a novel formulation of pairwise potentials that enables learning weights for different spatial ranges on a superpixel graph. The proposed C-CRF learning-based saliency model is tested on 6 benchmark datasets and compared with 11 existing methods. Our results and comparisons have provided further support to the proposed method in terms of precision-recall and F-measure. Furthermore, incorporating existing saliency models with pairwise cues through the C-CRF are shown to provide marked boosting performance over individual models.

Discriminative Training of Deep Fully Connected Continuous CRFs With Task-Specific Loss.

Recent works on deep conditional random fields (CRFs) have set new records on many vision tasks involving structured predictions. Here, we propose a fully connected deep continuous CRF model with task-specific losses for both discrete and continuous labeling problems. We exemplify the usefulness of the proposed model on multi-class semantic labeling (discrete) and the robust depth estimation (continuous) problems. In our framework, we model both the unary and the pairwise potential functions as deep convolutional neural networks (CNNs), which are jointly learned in an end-to-end fashion. The proposed method possesses the main advantage of continuously valued CRFs, which is a closed-form solution for the maximum a posteriori (MAP) inference. To better take into account the quality of the predicted estimates during the cause of learning, instead of using the commonly employed maximum likelihood CRF parameter learning protocol, we propose task-specific loss functions for learning the CRF parameters. It enables direct optimization of the quality of the MAP estimates during the learning process. Specifically, we optimize the multi-class classification loss for the semantic labeling task and the Tukey's biweight loss for the robust depth estimation problem. Experimental results on the semantic labeling and robust depth estimation tasks demonstrate that the proposed method compare favorably against both baseline and state-of-the-art methods. In particular, we show that although the proposed deep CRF model is continuously valued, with the equipment of task-specific loss, it achieves impressive results even on discrete labeling tasks.

A multimodal approach to estimating vigilance using EEG and forehead EOG

Objective. Covert aspects of ongoing user mental states provide key context information for user-aware human computer interactions. In this paper, we focus on the problem of estimating the vigilance of users using EEG and EOG signals. Approach. The PERCLOS index as vigilance annotation is obtained from eye tracking glasses. To improve the feasibility and wearability of vigilance estimation devices for real-world applications, we adopt a novel electrode placement for forehead EOG and extract various eye movement features, which contain the principal information of traditional EOG. We explore the effects of EEG from different brain areas and combine EEG and forehead EOG to leverage their complementary characteristics for vigilance estimation. Considering that the vigilance of users is a dynamic changing process because the intrinsic mental states of users involve temporal evolution, we introduce continuous conditional neural field and continuous conditional random field models to capture dynamic temporal dependency. Main results. We propose a multimodal approach to estimating vigilance by combining EEG and forehead EOG and incorporating the temporal dependency of vigilance into model training. The experimental results demonstrate that modality fusion can improve the performance compared with a single modality, EOG and EEG contain complementary information for vigilance estimation, and the temporal dependency-based models can enhance the performance of vigilance estimation. From the experimental results, we observe that theta and alpha frequency activities are increased, while gamma frequency activities are decreased in drowsy states in contrast to awake states. Significance. The forehead setup allows for the simultaneous collection of EEG and EOG and achieves comparative performance using only four shared electrodes in comparison with the temporal and posterior sites.

Fully Connected Continuous Conditional Random Field With Stochastic Cliques for Dark-Spot Detection In SAR Imagery

Dark-spot detection from synthetic aperture radar (SAR) imagery is a fundamental step in marine oil-spill detection and monitoring. However, achieving robust and accurate detection is difficult due to SAR sensor limitations and the complex marine environment. To address this problem, the large-scale spatial contextual information in SAR imagery has to be utilized to increase the class separability between the dark spot and the background. A stochastic fully connected continuous conditional random field (SFCCRF) approach to model SAR imagery and perform soft-label inference has been designed and built, leading to an efficient detection algorithm. Instead of treating all pixels in the imagery as being connected, SFCCRF determines the connectivity of two pixels in a stochastic manner based on their proximity in both feature space and image space. Since SFCCRF provides an efficient and effective way for modeling the large-scale spatial correlation effect, the resulting soft labels can resist the influence of speckle noise and highlight the difference between dark spot and the background. Dark-spot detection is achieved by binarizing the soft labels estimated by SFCCRF. The proposed algorithm is tested on both simulated and real SAR imageries. The results show that SFCCRF can delineate the dark spot with low commission and omission error rates.

Continuous Conditional Random Field Model for Predicting the Electrical Load of a Combined Cycle Power Plant

Existing power plants may consume significant amounts of fuel and require high operating costs, partly because of poor electrical power output estimates. This paper suggests a continuous conditional random field (C-CRF) model to predict more precisely the full-load electrical power output of a base load operated combined cycle power plant. We introduce three feature functions to model association potential and one feature function to model interaction potential. Together, these functions compose the C-CRF model, and the model is transformed into a multivariate Gaussian distribution with which the operation parameters can be modeled more efficiently. The performance of our model in estimating power output was evaluated by means of a real dataset and our model outperformed existing methods. Moreover, our model can be used to estimate confidence intervals of the predicted output and calculate several probabilities.