Class-conditional Probability Research Articles

Integrating Hierarchical Segmentation Maps With MRF Prior for Classification of Hyperspectral Images in a Bayesian Framework

This paper presents a new spectral-spatial classification method for hyperspectral (HS) images. The proposed method is based on integrating hierarchical segmentation results into Markov random field (MRF) spatial prior in the Bayesian framework. This work includes two main contributions. First, statistical region merging (SRM) segmentation algorithm is extended to a hierarchical version, HSRM. Second, a method for extracting a multilevel “fuzzy no-border/border” map from HSRM segmentation hierarchy is proposed, which are then exploited as weighting coefficients to modify the spatial prior of MRF-based multilevel logistic (MLL) model. The proposed method, named as MRF + HSRM, addresses the common problem of MRF-based methods, i.e., over-smoothing of classification result. Several experiments are conducted using real HS images to evaluate the performance of the proposed method in comparison with conventional MRF, and some state-of-the-art weighted MRF and object-based classifiers. To estimate the class conditional probability distribution in Bayesian framework, probabilistic support vector machine (SVM) and subspace multinomial logistic regression (MLRsub) classifiers are used. The experimental results demonstrate that the proposed method is able to generate more homogeneous regions similar to MRF-based methods, while preserving class boundaries as accurately as segmentation-based methods. The overhead computational burden of the proposed hierarchical segmentation stage is negligible considering the improvement it offers in classification results.

Automated segmentation of the thyroid gland on thoracic CT scans by multiatlas label fusion and random forest classification.

The thyroid is an endocrine gland that regulates metabolism. Thyroid image analysis plays an important role in both diagnostic radiology and radiation oncology treatment planning. Low tissue contrast of the thyroid relative to surrounding anatomic structures makes manual segmentation of this organ challenging. This work proposes a fully automated system for thyroid segmentation on CT imaging. Following initial thyroid segmentation with multiatlas joint label fusion, a random forest (RF) algorithm was applied. Multiatlas label fusion transfers labels from labeled atlases and warps them to target images using deformable registration. A consensus atlas solution was formed based on optimal weighting of atlases and similarity to a given target image. Following the initial segmentation, a trained RF classifier employed voxel scanning to assign class-conditional probabilities to the voxels in the target image. Thyroid voxels were categorized with positive labels and nonthyroid voxels were categorized with negative labels. Our method was evaluated on CT scans from 66 patients, 6 of which served as atlases for multiatlas label fusion. The system with independent multiatlas label fusion method and RF classifier achieved average dice similarity coefficients of [Formula: see text] and [Formula: see text], respectively. The system with sequential multiatlas label fusion followed by RF correction increased the dice similarity coefficient to [Formula: see text] and improved the segmentation accuracy.

Non-stationary feature fusion of face and palmprint multimodal biometrics

Multimodal biometrics provides high performance in biometric recognition systems with respect to unibiometric systems as they offer a more universal approach, added security and better recognition accuracy. Moreover, data acquisition at the feature level brings out rich information from the traits, thus fusion of modalities at this level is desirable. In this paper we propose a novel fusion technique called non-stationary feature fusion where a new structure of interleaved matrix is constructed using local features extracted from two modalities i.e. face and palmprint images. A block based Discrete Cosine Transform (DCT) algorithm is used to construct a fused feature vector by extracting independent feature vectors from each spatial image. This fused feature vector contains nonlinear information that is used to train a Gaussian Mixture Models (GMM) based statistical model. The model provides accurate estimation of the class conditional probability density function of the fused feature vector. The proposed method produces recognition rates as high as 99.7% and 97% when tested on benchmark databases-ORL-PolyU and FERET-PolyU respectively. These rates are achieved using 23% low frequency DCT coefficients. The new technique is shown to outperform existing feature level fusion methods including methods based on matching and decision level fusion.

The Parzen Window method: In terms of two vectors and one matrix

Pattern classification methods assign an object to one of several predefined classes/categories based on features extracted from observed attributes of the object (pattern). When L discriminatory features for the pattern can be accurately determined, the pattern classification problem presents no difficulty. However, precise identification of the relevant features for a classification algorithm (classifier) to be able to categorize real world patterns without errors is generally infeasible. In this case, the pattern classification problem is often cast as devising a classifier that minimizes the misclassification rate. One way of doing this is to consider both the pattern attributes and its class label as random variables, estimate the posterior class probabilities for a given pattern and then assign the pattern to the class/category for which the posterior class probability value estimated is maximum. More often than not, the form of the posterior class probabilities is unknown.The so-called Parzen Window approach is widely employed to estimate class-conditional probability (class-specific probability) densities for a given pattern. These probability densities can then be utilized to estimate the appropriate posterior class probabilities for that pattern. However, the Parzen Window scheme can become computationally impractical when the size of the training dataset is in the tens of thousands and L is also large (a few hundred or more). Over the years, various schemes have been suggested to ameliorate the computational drawback of the Parzen Window approach, but the problem still remains outstanding and unresolved.In this paper, we revisit the Parzen Window technique and introduce a novel approach that may circumvent the aforementioned computational bottleneck. The current paper presents the mathematical aspect of our idea. Practical realizations of the proposed scheme will be given elsewhere.

Modified criterion to select useful unlabeled data for improving semi-supervised support vector machines

Recent studies have demonstrated that semi-supervised learning (SSL) approaches that use both labeled and unlabeled data are more effective and robust than those that use only labeled data. In SemiBoost, a boosting framework for SSL, a similarity based criterion is developed to select (and utilize) a small amount of useful unlabeled data. However, sometimes it does not work appropriately, particularly when the unlabeled data are near the boundary. In order to address this concern, in this paper the selection criterion is modified using the class-conditional probability in addition to the similarity: first, the criterion is decomposed into three terms of positive class term, negative class term, and unlabeled term; second, when computing the confidences of unlabeled data, using the conditional probability estimated, impacts of the three terms on the confidences are adjusted; third, some unlabeled data that have higher confidences are selected and, together with labeled data, used for re-training a supervised classifier. This select-and-train process is repeated until a termination condition is met. The experimental results, obtained using semi-supervised support vector machines (S3VMs) with benchmark data, demonstrate that the proposed algorithm can compensate for the shortcomings of the traditional S3VMs and, when compared with previous approaches, can achieve further improved results in terms of the classification accuracy.

Neural network based class-conditional probability density function using kernel trick for supervised classifier

The practical limitation of the Bayes classifier used in pattern recognition is computing the class-conditional probability density function (pdf) of the vectors belonging to the corresponding classes. In this paper, a neural network based approach is proposed to model the class-conditional pdf, which can be further used in supervised classifier (e.g. Bayes classifier). It is also suggested to use kernel version (using kernel trick) of the proposed approach to obtain the class-conditional pdf of the corresponding training vectors in the higher dimensional space. This is used for better class separation and hence better classification rate is achieved. The performance of the proposed technique is validated by using the synthetic data and the real data. The simulation results show that the proposed technique on synthetic data and the real data performs well (in terms of classification accuracy) when compared with the classical Fisher׳s Linear Discriminant Analysis (LDA) and Gaussian based Kernel-LDA.

Experimental study on generalization capability of extended naive Bayesian classifier

Extended naive Bayesian classifier (ENBC) is a general framework of NBCs, which is developed based on \(t\)-norm based ordered weighted averaging (\(t\)-OWA) operator and uses the weighted summation of products of margin probabilities to determine class-conditional probability. Since ENBC was proposed in 2006, there is no such a study which tests the performances of ENBC on the real classification datasets. Thus, in this paper we conduct an experimental investigation to ENBC’s generalization capability based on 44 benchmark KEEL and UCI datasets. The analysis shows that (1) ENBC is instable and its aggregation weights are sensitive to the order of training samples and (2) ENBC indeed has higher generalization capability than the existing NBCs, e.g., normal naive Bayesian and flexible naive Bayesian when its weight is properly selected.

Crop Mapping of Complex Agricultural Landscapes Based on Discriminant Space

Crop mapping is related to planting conditions, remote sensing data source, and performances of various classifiers. Considering the above key factors, this study discusses a crop mapping approach for complex agricultural landscapes based on a conceptual frame that is called discriminant space. The introduced discriminant space-based model follows the pattern and process rules in geosciences and is found by the variables closely related to crop growth. On this basis, a method of classification based on probability calibration algorithm (CPCA) is developed, which consists of several steps. First, discriminant variables were extracted and then the experimental discriminant space was founded. Spectral analysis was carried out in this process to select the dominant impact factors for crop mapping. Based on the founded discriminant space, we calculated the class-conditional probabilities for each category using kernel density estimation and separated the residuals of class-conditional probabilities from the structural component through a trend-surface analysis. Then, calibration of the class-conditional probabilities was implemented using the predicted residuals generated by simple kriging with varying local means. Classification was achieved according to the calibrated probabilities with respect to each category. Accuracy assessment and kappa analysis were used to evaluate the identification results. Statistical test procedure was applied to measure the differences between classification maps and the validation data based on individual categories. Results are discussed for an intensively cropped area in south China. Compared with two other supervised classifiers, the overall accuracy of the proposal method increases by more than 10% and is able to achieve 87.12%. Moreover, there are some substantial increases in the producer's accuracy and the user's accuracy of single-cropping rice and late rice, with a precision increase of more than 15%, effectively improves the identification accuracy of rice in the research region. The proposed method overcomes the limitations due to spectrum characteristics and considers the structural and the random properties of spatial distribution for categories in local regions. Therefore, a similar operation can usually be implemented for crop identification of complex agricultural landscapes.

Mapping model validation metrics to subject matter expert scores for model adequacy assessment

This paper develops a novel approach to incorporate the contributions of both quantitative validation metrics and qualitative subject matter expert (SME) evaluation criteria in model validation assessment. The relationship between validation metrics (input) and SME scores (output) is formulated as a classification problem, and a probabilistic neural network (PNN) is constructed to execute this mapping. Establishing PNN classifiers for a wide variety of combinations of validation metrics allows for a quantitative comparison of validation metric performance in representing SME judgment. An advantage to this approach is that it semi-automates the model validation process and subsequently is capable of incorporating the contributions of large data sets of disparate response quantities of interest in model validation assessment. The effectiveness of this approach is demonstrated on a complex real-world problem involving the shock qualification testing of a floating shock platform. A data set of experimental and simulated pairs of time history comparisons along with associated SME scores and computed validation metrics is obtained and utilized to construct the PNN classifiers through K-fold cross validation. A wide range of validation metrics for time history comparisons is considered including feature-specific metrics (phase and magnitude error), a frequency metric (shock response spectra), a time-frequency metric (wavelet decomposition), and a global metric (index of agreement). The PNN classifiers constructed using a Parzen kernel for the class conditional probability density function whose smoothing parameter is optimized using a genetic algorithm performs well in representing SME judgment.

Application of random forests methods to diabetic retinopathy classification analyses.

BackgroundDiabetic retinopathy (DR) is one of the leading causes of blindness in the United States and world-wide. DR is a silent disease that may go unnoticed until it is too late for effective treatment. Therefore, early detection could improve the chances of therapeutic interventions that would alleviate its effects.MethodologyGraded fundus photography and systemic data from 3443 ACCORD-Eye Study participants were used to estimate Random Forest (RF) and logistic regression classifiers. We studied the impact of sample size on classifier performance and the possibility of using RF generated class conditional probabilities as metrics describing DR risk. RF measures of variable importance are used to detect factors that affect classification performance.Principal FindingsBoth types of data were informative when discriminating participants with or without DR. RF based models produced much higher classification accuracy than those based on logistic regression. Combining both types of data did not increase accuracy but did increase statistical discrimination of healthy participants who subsequently did or did not have DR events during four years of follow-up. RF variable importance criteria revealed that microaneurysms counts in both eyes seemed to play the most important role in discrimination among the graded fundus variables, while the number of medicines and diabetes duration were the most relevant among the systemic variables.Conclusions and SignificanceWe have introduced RF methods to DR classification analyses based on fundus photography data. In addition, we propose an approach to DR risk assessment based on metrics derived from graded fundus photography and systemic data. Our results suggest that RF methods could be a valuable tool to diagnose DR diagnosis and evaluate its progression.

Probabilistic Novelty Detection With Support Vector Machines

Novelty detection, or one-class classification, is of particular use in the analysis of high-integrity systems, in which examples of failure are rare in comparison with the number of examples of stable behaviour, such that a conventional multi-class classification approach cannot be taken. Support Vector Machines (SVMs) are a popular means of performing novelty detection, and it is conventional practice to use a train-validate-test approach, often involving cross-validation, to train the one-class SVM, and then select appropriate values for its parameters. An alternative method, used with multi-class SVMs, is to calibrate the SVM output into conditional class probabilities. A probabilistic approach offers many advantages over the conventional method, including the facility to select automatically a probabilistic novelty threshold. The contributions of this paper are (i) the development of a probabilistic calibration technique for one-class SVMs, such that on-line novelty detection may be performed in a probabilistic manner; and (ii) the demonstration of the advantages of the proposed method (in comparison to the conventional one-class SVM methodology) using case studies, in which one-class probabilistic SVMs are used to perform condition monitoring of a high-integrity industrial combustion plant, and in detecting deterioration in patient physiological condition during patient vital-sign monitoring.

An NMF-framework for Unifying Posterior Probabilistic Clustering and Probabilistic Latent Semantic Indexing

In document clustering, a document may be assigned to multiple clusters and the probabilities of a document belonging to different clusters are directly normalized. We propose a new Posterior Probabilistic Clustering (PPC) model that has this normalization property. The clustering model is based on Nonnegative Matrix Factorization (NMF) and flexible such that if we use class conditional probability normalization, the model reduces to Probabilistic Latent Semantic Indexing (PLSI). Systematic comparison and evaluation indicates that PPC is competitive with other state-of-art clustering methods. Furthermore, the results of PPC are more sparse and orthogonal, both of which are highly desirable.

Probability-enhanced sufficient dimension reduction for binary classification.

In high-dimensional data analysis, it is of primary interest to reduce the data dimensionality without loss of information. Sufficient dimension reduction (SDR) arises in this context, and many successful SDR methods have been developed since the introduction of sliced inverse regression (SIR) [Li (1991) Journal of the American Statistical Association 86, 316-327]. Despite their fast progress, though, most existing methods target on regression problems with a continuous response. For binary classification problems, SIR suffers the limitation of estimating at most one direction since only two slices are available. In this article, we develop a new and flexible probability-enhanced SDR method for binary classification problems by using the weighted support vector machine (WSVM). The key idea is to slice the data based on conditional class probabilities of observations rather than their binary responses. We first show that the central subspace based on the conditional class probability is the same as that based on the binary response. This important result justifies the proposed slicing scheme from a theoretical perspective and assures no information loss. In practice, the true conditional class probability is generally not available, and the problem of probability estimation can be challenging for data with large-dimensional inputs. We observe that, in order to implement the new slicing scheme, one does not need exact probability values and the only required information is the relative order of probability values. Motivated by this fact, our new SDR procedure bypasses the probability estimation step and employs the WSVM to directly estimate the order of probability values, based on which the slicing is performed. The performance of the proposed probability-enhanced SDR scheme is evaluated by both simulated and real data examples.

Sequential pattern recognition by maximum conditional informativity

Sequential pattern recognition assumes the features to be measured successively, one at a time, and therefore the key problem is to choose the next feature optimally. However, the choice of the features may be strongly influenced by the previous feature measurements and therefore the on-line ordering of features is difficult. There are numerous methods to estimate class-conditional probability distributions but it is usually computationally intractable to derive the corresponding conditional marginals. In literature there is no exact method of on-line feature ordering except for the strongly simplifying naive Bayes models. We show that the problem of sequential recognition has an explicit analytical solution which is based on approximation of the class-conditional distributions by mixtures of product components. As the marginal distributions of product mixtures are directly available by omitting superfluous terms in the products, we have a unique non-trivial possibility to evaluate at any decision level the conditional informativity of unobserved features for a general problem of statistical recognition. In this way the most informative feature guarantees, for any given set of preceding measurements, the maximum decrease of decision uncertainty.

Selecting One Dependency Estimators in Bayesian Network Using Different MDL Scores and Overfitting Criterion

The Averaged One Dependency Estimator (AODE) is integrated all possible Super-Parent-One-Dependency Estimators (SPODEs) and estimates class conditional probabilities by averaging them. In an AODE network some redundant SPODEs maybe result in some bias of classifiers, as a consequence, it could reduce the classification accuracy substantially. In this paper, a kind of MDL metrics is used to select SPODEs in a whole or partially, therefore there are three different classifiers presented. The performance comparisons between them and AODE have been shown not only the theoretical analyses are reasonable, but also efficient and effective. And Mean Square Error (MSE) is used to test overfitting. Experiential results have indicated that the classifier using MDL score metrics had better performance than original AODE, and at the same time, has less overfitting. At the end of the paper, further discussions and verifications of some properties of overfitting have also shown in the experiments.

Generalized Bhattacharyya and Chernoff upper bounds on Bayes error using quasi-arithmetic means

Bayesian classification labels observations based on given prior information, namely class-a priori and class-conditional probabilities. Bayes’ risk is the minimum expected classification cost that is achieved by the Bayes’ test, the optimal decision rule. When no cost incurs for correct classification and unit cost is charged for misclassification, Bayes’ test reduces to the maximum a posteriori decision rule, and Bayes risk simplifies to Bayes’ error, the probability of error. Since calculating this probability of error is often intractable, several techniques have been devised to bound it with closed-form formula, introducing thereby measures of similarity and divergence between distributions like the Bhattacharyya coefficient and its associated Bhattacharyya distance. The Bhattacharyya upper bound can further be tightened using the Chernoff information that relies on the notion of best error exponent. In this paper, we first express Bayes’ risk using the total variation distance on scaled distributions. We then elucidate and extend the Bhattacharyya and the Chernoff upper bound mechanisms using generalized weighted means. We provide as a byproduct novel notions of statistical divergences and affinity coefficients. We illustrate our technique by deriving new upper bounds for the univariate Cauchy and the multivariate t-distributions, and show experimentally that those bounds are not too distant to the computationally intractable Bayes’ error.

A Robust Probability Classifier Based on the Modifiedχ2-Distance

We propose a robust probability classifier model to address classification problems with data uncertainty. A class-conditional probability distributional set is constructed based on the modifiedχ2-distance. Based on a “linear combination assumption” for the posterior class-conditional probabilities, we consider a classification criterion using the weighted sum of the posterior probabilities. An optimal robust minimax classifier is defined as the one with the minimal worst-case absolute error loss function value over all possible distributions belonging to the constructed distributional set. Based on the conic duality theorem, we show that the resulted optimization problem can be reformulated into a second order cone programming problem which can be efficiently solved by interior algorithms. The robustness of the proposed model can avoid the “overlearning” phenomenon on training sets and thus keep a comparable accuracy on test sets. Numerical experiments validate the effectiveness of the proposed model and further show that it also provides promising results on multiple classification problems.

A speaker invariant speech recognition technique using HFCC features in isolated Hindi words

A speaker invariant speech recognition system is proposed by analysing the characteristics of speech signal. The distinctive features are derived from the speech data using discrete wavelet transforms (DWT) and human factor cepestral coefficient (HFCC) technique. This HFCC technique provides an immense impact on signal decoupling process for adjusting parameters in noise smoothing and spectral resolution. We have created a speech repository of 12 isolated Hindi words. The principal component analysis (PCA) is applied on speech features obtained from HFCC analysis in order to reduce the dimension of feature space. We have applied Bayes' decision rule for classification with multivariate normal distribution which follows the class conditional probability density function for each training classes. The performance of the classifier has been evaluated by calculating the misclassification error probability. Experimental results of proposed method are analysed and compared with the existing methods like MFCC with DWT, MFCC with PCA, DWT with PCA, etc. We have achieved promising classification results using HFCC-based speech features for speaker invariant speech identification system.

A Bayesian Model for Crowd Escape Behavior Detection

People naturally escape from a place when unexpected events happen. Based on this observation, efficient detection of crowd escape behavior in surveillance videos is a promising way to perform timely detection of anomalous situations. In this paper, we propose a Bayesian framework for escape detection by directly modeling crowd motion in both the presence and absence of escape events. Specifically, we introduce the concepts of potential destinations and divergent centers to characterize crowd motion in the above two cases respectively, and construct the corresponding class-conditional probability density functions of optical flow. Escape detection is finally performed based on the proposed Bayesian framework. Although only data associated with nonescape behavior are included in the training set, the density functions associated with the case of escape can also be adaptively updated using observed data. In addition, the identified divergent centers indicate possible locations at which the unexpected events occur. The performance of our proposed method is validated in a number of experiments on crowd escape detection in various scenarios.

SUPERVISED CLASSIFICATION OF POLARIMETRIC SAR IMAGERY USING TEMPORAL AND CONTEXTUAL INFORMATION

Abstract. Using the context as a source of ancillary information in classification process provides a powerful tool to obtain better class discrimination. Modelling context using Markov Random Fields (MRFs) and combining with Bayesian approach, a context-based supervised classification method is proposed. In this framework, to have a full use of the statistical a priori knowledge of the data, the spatial relation of the neighbouring pixels was used. The proposed context-based algorithm combines a Gaussian-based wishart distribution of PolSAR images with temporal and contextual information. This combination was done through the Bayes decision theory: the class-conditional probability density function and the prior probability are modelled by the wishart distribution and the MRF model. Given the complexity and similarity of classes, in order to enhance the class separation, simultaneously two PolSAR images from two different seasons (leaf-on and leaf-off) were used. According to the achieved results, the maximum improvement in the overall accuracy of classification using WMRF (Combining Wishart and MRF) compared to the wishart classifier when the leaf-on image was used. The highest accuracy obtained was when using the combined datasets. In this case, the overall accuracy of the wishart and WMRF methods were 72.66% and 78.95% respectively.