Prior Class Probabilities Research Articles

Adaptive Texture Description and Estimation of the Class Prior Probabilities for Seminal Quality Control

Motivation of the Thesis Semen quality assessment is a crucial task in artificial insemination (AI) processes, both human and animal. Animal AI allows farmers to save time and money (e.g. working with a limited number of animals). They purchase semen samples to companies, which have to carry out strict quality controls to guarantee that they are optimal for fertilization. A sample with a high proportion of (i) dead spermatozoa, or (ii) sperm heads with damaged acrosomes (the acrosome is a membrane that covers the anterior part of the sperm head and makes possible the penetration into the ovum) will have low fertilization potential. Therefore, sperm vitality and acrosome integrity are two of the parameters assessed by veterinaries in semen quality control processes. Both are assessed by means of a visual process which entails expensive equipments (stains and fluorescence microscopes) and may be a source of errors, as any manual process is. The contributions in the field of Image Processing and Machine Learning made on this PhD. Thesis [2] may be used to develop an automatic process to assess the proportions of damaged acrosomes or dead spermatozoa using just a phase contrast microscope (which almost any lab has) and a digital camera. Concretely, several texture description approaches have been evaluated. In addition, a new intelligent segmentation process, an adaptive texture description method, and two robust approaches for estimating class proportions of unlabelled datasets have been proposed. All these methods are applied to automatic boar semen quality estimation.

Effect of separate sampling on classification accuracy

Measurements are commonly taken from two phenotypes to build a classifier, where the number of data points from each class is predetermined, not random. In this 'separate sampling' scenario, the data cannot be used to estimate the class prior probabilities. Moreover, predetermined class sizes can severely degrade classifier performance, even for large samples. We employ simulations using both synthetic and real data to show the detrimental effect of separate sampling on a variety of classification rules. We establish propositions related to the effect on the expected classifier error owing to a sampling ratio different from the population class ratio. From these we derive a sample-based minimax sampling ratio and provide an algorithm for approximating it from the data. We also extend to arbitrary distributions the classical population-based Anderson linear discriminant analysis minimax sampling ratio derived from the discriminant form of the Bayes classifier. All the codes for synthetic data and real data examples are written in MATLAB. A function called mmratio, whose output is an approximation of the minimax sampling ratio of a given dataset, is also written in MATLAB. All the codes are available at: http://gsp.tamu.edu/Publications/supplementary/shahrokh13b.

Class distribution estimation based on the Hellinger distance

Class distribution estimation (quantification) plays an important role in many practical classification problems. Firstly, it is important in order to adapt the classifier to the operational conditions when they differ from those assumed in learning. Additionally, there are some real domains where the quantification task is itself valuable due to the high variability of the class prior probabilities. Our novel quantification approach for two-class problems is based on distributional divergence measures. The mismatch between the test data distribution and validation distributions generated in a fully controlled way is measured by the Hellinger distance in order to estimate the prior probability that minimizes this divergence. Experimental results on several binary classification problems show the benefits of this approach when compared to such approaches as counting the predicted class labels and other methods based on the classifier confusion matrix or on posterior probability estimations. We also illustrate these techniques as well as their robustness against the base classifier performance (a neural network) with a boar semen quality control setting. Empirical results show that the quantification can be conducted with a mean absolute error lower than 0.008, which seems very promising in this field.

Hybrid Bayesian Classifier for Improved Classification Accuracy

The widely used Bayesian classifier is based on the assumption of equal prior probabilities for all the classes. However, inclusion of equal prior probabilities may not guarantee high classification accuracy for the individual classes. Here, we propose a novel technique-Hybrid Bayesian Classifier (HBC)-where the class prior probabilities are determined by unmixing a supplemental low spatial-high spectral resolution multispectral (MS) data that are assigned to every pixel in a high spatial-low spectral resolution MS data in Bayesian classification. This is demonstrated with two separate experiments-first, class abundances are estimated per pixel by unmixing Moderate Resolution Imaging Spectroradiometer data to be used as prior probabilities, while posterior probabilities are determined from the training data obtained from ground. These have been used for classifying the Indian Remote Sensing Satellite LISS-III MS data through Bayesian classifier. In the second experiment, abundances obtained by unmixing Landsat Enhanced Thematic Mapper Plus are used as priors, and posterior probabilities are determined from the ground data to classify IKONOS MS images through Bayesian classifier. The results indicated that HBC systematically exploited the information from two image sources, improving the overall accuracy of LISS-III MS classification by 6% and IKONOS MS classification by 9%. Inclusion of prior probabilities increased the average producer's and user's accuracies by 5.5% and 6.5% in case of LISS-III MS with six classes and 12.5% and 5.4% in IKONOS MS for five classes considered.

Multiclass Common Spatial Patterns and Information Theoretic Feature Extraction

We address two shortcomings of the common spatial patterns (CSP) algorithm for spatial filtering in the context of brain--computer interfaces (BCIs) based on electroencephalography/magnetoencephalography (EEG/MEG): First, the question of optimality of CSP in terms of the minimal achievable classification error remains unsolved. Second, CSP has been initially proposed for two-class paradigms. Extensions to multiclass paradigms have been suggested, but are based on heuristics. We address these shortcomings in the framework of information theoretic feature extraction (ITFE). We show that for two-class paradigms, CSP maximizes an approximation of mutual information of extracted EEG/MEG components and class labels. This establishes a link between CSP and the minimal classification error. For multiclass paradigms, we point out that CSP by joint approximate diagonalization (JAD) is equivalent to independent component analysis (ICA), and provide a method to choose those independent components (ICs) that approximately maximize mutual information of ICs and class labels. This eliminates the need for heuristics in multiclass CSP, and allows incorporating prior class probabilities. The proposed method is applied to the dataset IIIa of the third BCI competition, and is shown to increase the mean classification accuracy by 23.4% in comparison to multiclass CSP.

Localized empirical discriminant analysis

Some empirical localized discriminant analysis methods for classifying images are introduced. They use spatial correlation of images in order to improve classification reducing the ‘pseudo-nuisance’ present in pixel-wise discriminant analysis. The result is obtained through an empirical (data driven) and local (pixel-wise) choice of the prior class probabilities. Local empirical discriminant analysis is formalized in a framework that focuses on the concept of visibility of a class that is introduced. Numerical experiments are performed on synthetic and real data. In particular, methods are applied to the problem of retrieving the cloud mask from remotely sensed images. In both cases classical and new local discriminant methods are compared to the ICM method.

Efficient approximate leave-one-out cross-validation for kernel logistic regression

Kernel logistic regression (KLR) is the kernel learning method best suited to binary pattern recognition problems where estimates of a-posteriori probability of class membership are required. Such problems occur frequently in practical applications, for instance because the operational prior class probabilities or equivalently the relative misclassification costs are variable or unknown at the time of training the model. The model parameters are given by the solution of a convex optimization problem, which may be found via an efficient iteratively re-weighted least squares (IRWLS) procedure. The generalization properties of a kernel logistic regression machine are however governed by a small number of hyper-parameters, the values of which must be determined during the process of model selection. In this paper, we propose a novel model selection strategy for KLR, based on a computationally efficient closed-form approximation of the leave-one-out cross-validation procedure. Results obtained on a variety of synthetic and real-world benchmark datasets are given, demonstrating that the proposed model selection procedure is competitive with a more conventional k-fold cross-validation based approach and also with Gaussian process (GP) classifiers implemented using the Laplace approximation and via the Expectation Propagation (EP) algorithm.

An Equalized Heteroscedastic Linear Discriminant Analysis Algorithm

Heteroscedastic linear discriminant analysis (HLDA) is a widely used feature extraction algorithm. This method, however, suffers from unbalanced training data in some cases. In this letter, we equalize the objective function and statistics of HLDA and present an equalized HLDA algorithm, which balances the training data according to the class prior probability. Simulations as well as experimental results for the task of language identification are used to demonstrate the effectiveness of the proposed method.

Classification based upon gene expression data: bias and precision of error rates

Gene expression data offer a large number of potentially useful predictors for the classification of tissue samples into classes, such as diseased and non-diseased. The predictive error rate of classifiers can be estimated using methods such as cross-validation. We have investigated issues of interpretation and potential bias in the reporting of error rate estimates. The issues considered here are optimization and selection biases, sampling effects, measures of misclassification rate, baseline error rates, two-level external cross-validation and a novel proposal for detection of bias using the permutation mean. Reporting an optimal estimated error rate incurs an optimization bias. Downward bias of 3-5% was found in an existing study of classification based on gene expression data and may be endemic in similar studies. Using a simulated non-informative dataset and two example datasets from existing studies, we show how bias can be detected through the use of label permutations and avoided using two-level external cross-validation. Some studies avoid optimization bias by using single-level cross-validation and a test set, but error rates can be more accurately estimated via two-level cross-validation. In addition to estimating the simple overall error rate, we recommend reporting class error rates plus where possible the conditional risk incorporating prior class probabilities and a misclassification cost matrix. We also describe baseline error rates derived from three trivial classifiers which ignore the predictors. R code which implements two-level external cross-validation with the PAMR package, experiment code, dataset details and additional figures are freely available for non-commercial use from http://www.maths.qut.edu.au/profiles/wood/permr.jsp

A unified framework for MAP estimation in remote sensing image segmentation

A complete framework is proposed for applying the maximum a posteriori (MAP) estimation principle in remote sensing image segmentation. The MAP principle provides an estimate for the segmented image by maximizing the posterior probabilities of the classes defined in the image. The posterior probability can be represented as the product of the class conditional probability (CCP) and the class prior probability (CPP). In this paper, novel supervised algorithms for the CCP and the CPP estimations are proposed which are appropriate for remote sensing images where the estimation process might to be done in high-dimensional spaces. For the CCP, a supervised algorithm which uses the support vector machines (SVM) density estimation approach is proposed. This algorithm uses a novel learning procedure, derived from the main field theory, which avoids the (hard) quadratic optimization problem arising from the traditional formulation of the SVM density estimation. For the CPP estimation, Markov random field (MRF) is a common choice which incorporates contextual and geometrical information in the estimation process. Instead of using predefined values for the parameters of the MRF, an analytical algorithm is proposed which automatically identifies the values of the MRF parameters. The proposed framework is built in an iterative setup which refines the estimated image to get the optimum solution. Experiments using both synthetic and real remote sensing data (multispectral and hyperspectral) show the powerful performance of the proposed framework. The results show that the proposed density estimation algorithm outperforms other algorithms for remote sensing data over a wide range of spectral dimensions. The MRF modeling raises the segmentation accuracy by up to 10% in remote sensing images.

Comparison of different methodological implementations of voxel-based morphometry in neurodegenerative disease

Voxel-based morphometry (VBM) is a popular method for probing inter-group differences in brain morphology. Variation in the detailed implementation of the algorithm, however, will affect the apparent results of VBM analyses and in turn the inferences drawn about the anatomic expression of specific disease states. We qualitatively assessed group comparisons of 43 normal elderly control subjects and 51 patients with probable Alzheimer's disease, using five different VBM variations. Based on the known pathologic expression of the disease, we evaluated the biological plausibility of each. The use of a custom template and custom tissue class prior probability images (priors) produced inter-group comparison maps with greater biological plausibility than the use of the Montreal Neurological Institute (MNI) template and priors. We present a method for initializing the normalization to a custom template, and conclude that, when incorporated into the VBM processing chain, it yields the most biologically plausible inter-group differences of the five methods presented.

Target classification approach based on the belief function theory

A theoretical framework is presented for target classification based on the belief theory on the continuous space. The proposed approach is applicable when class-conditioned densities of feature/attribute measurements are known only partially, as subjective models of a potential betting behaviour. Prior class probabilities may also be unknown. Numerical examples are provided to illustrate how the proposed approach is more cautious in decision making and produces very different results from those obtained using the Bayesian classifier.

Stochastic simulation of large grids using free and public domain software

This paper proposes a tiled map procedure enabling sequential indicator simulation on grids consisting of several tens of millions of cells, without putting excessive memory requirements. Spatial continuity across map tiles is handled by conditioning adjacent tiles on their shared boundaries. Tiles across the area can be characterized by dissimilar models of spatial continuity (semi-variograms) thus relieving the requirement of a global stationarity decision. Additionally, the approach provides a simple mechanism for reseeding the pseudo random number generator. Implementation of the algorithm involved small modifications to a GSLIB program and Bash and awk scripting. The software was stable on several platforms, including 32-bit systems with a 4 Gb memory addressing limit. In an experiment we simulated 25 realizations of a 11,274×13,000 grid representing local uncertainty in the Dutch land cover at 25 m resolution. With the objective of mimicking the typical absence of well-distributed hard reference data, the simulations were only conditioned on local prior class probabilities and semi-variograms. Output was evaluated on the basis of reproduction of target levels of (1) cover type proportions, (2) overall class label accuracy and (3) spatially averaged local Shannon entropy. As expected, the realized statistics differed significantly from the target levels. However, the differences were consistent over the borders and the insides of map tiles. Thus, they did not result from the tiled map procedure but rather should be attributed to the used semi-conditional sequential indicator simulator. The current implementation can easily be adapted to accept other simulation algorithms.

CLASSIFICATION OF MULTITEMPORAL REMOTE-SENSING IMAGES BY A FUZZY FUSION OF SPECTRAL AND SPATIO-TEMPORAL CONTEXTUAL INFORMATION

A fuzzy-logic approach to the classification of multitemporal, multisensor remote-sensing images is proposed. The approach is based on a fuzzy fusion of three basic sources of information: spectral, spatial and temporal contextual information sources. It aims at improving the accuracy over that of single-time noncontextual classification. Single-time class posterior probabilities, which are used to represent spectral information, are estimated by Multilayer Perceptron neural networks trained for each single-time image, thus making the approach applicable to multisensor data. Both the spatial and temporal kinds of contextual information are derived from the single-time classification maps obtained by the neural networks. The expert's knowledge of possible transitions between classes at two different times is exploited to extract temporal contextual information. The three kinds of information are then fuzzified in order to apply a fuzzy reasoning rule for their fusion. Fuzzy reasoning is based on the "MAX" fuzzy operator and on information about class prior probabilities. Finally, the class with the largest fuzzy output value is selected for each pixel in order to provide the final classification map. Experimental results on a multitemporal data set consisting of two multisensor (Landsat TM and ERS-1 SAR) images are reported. The accuracy of the proposed fuzzy spatio-temporal contextual classifier is compared with those obtained by the Multilayer Perceptron neural networks and a reference classification approach based on Markov Random Fields (MRFs). Results show the benefit of adding spatio-temporal contextual information to the classification scheme, and suggest that the proposed approach represents an interesting alternative to the MRF-based approach, in particular, in terms of simplicity.

The influence of prior knowledge on the expected performance of a classifier

In this paper, we study the probabilistic properties of pattern classifiers in discrete feature space. The principle of Bayesian averaging of recognition performance is used for this analysis. We consider two cases: (a) prior probabilities of classes are unknown, and (b) prior probabilities of classes are known. The misclassification probability is represented as a random value, for which the characteristic function (expressed via Kummer hypergeometric function) and absolute moments are analytically derived. For the case of unknown priors, an approximate formula for calculation of sufficient learning sample size is obtained. The comparison between the performances for two considered cases is made. As an example, we consider the problem of mutational hotspots classification in genetic sequences.

Improved classification of Landsat Thematic Mapper data using modified prior probabilities in large and complex landscapes

The use of modified prior probabilities to exploit ancillary data and increase classification accuracy has been proposed before. However, this method has not been widely applied because it has heavy computing requirements and because obtaining prior probability estimates has presented practical problems. This article presents a procedure that generates large sets of prior probability estimates from class frequencies modelled with ancillary data and a Mahalanobis Distance selection of previously classified pixels. The method produces a pixel sample size that is large enough to estimate class frequencies in numerous strata, which is particularly desirable for the study of large and complex landscapes. A case study is presented in which the procedure made it possible to estimate 537 sets of prior probabilities for an entire Landsat Thematic Mapper (TM) scene of central Costa Rica. After modifying the class prior probabilities, the overall classification consistency of the training sites improved from 74.6% to 91.9%, while the overall classification accuracy of sites controlled in the field by independent studies improved from 68.7% to 89.0%. The classification accuracy was most improved for spectrally similar classes. The method improves classification accuracy in large and complex landscapes with spectrally mixed land-cover categories.

The class imbalance problem: A systematic study1

In machine learning problems, differences in prior class probabilities -- or class imbalances -- have been reported to hinder the performance of some standard classifiers, such as decision trees. This paper presents a systematic study aimed at answering three different questions. First, we attempt to understand the nature of the class imbalance problem by establishing a relationship between concept complexity, size of the training set and class imbalance level. Second, we discuss several basic re-sampling or cost-modifying methods previously proposed to deal with the class imbalance problem and compare their effectiveness. The results obtained by such methods on artificial domains are linked to results in real-world domains. Finally, we investigate the assumption that the class imbalance problem does not only affect decision tree systems but also affects other classification systems such as Neural Networks and Support Vector Machines.

A neural-statistical approach to multitemporal and multisource remote-sensing image classification

A data fusion approach to the classification of multisource and multitemporal remote-sensing images is proposed. The method is based on the application of the Bayes rule for minimum error to the compound classification of pairs of multisource images acquired at two different dates. In particular, the fusion of multisource data is obtained by using multilayer perceptron neural networks for a nonparametric estimation of posterior class probabilities. The temporal correlation between images is taken into account by the prior joint probabilities of classes at the two dates. As a novel contribution of this paper, such joint probabilities are automatically estimated by applying a specific formulation of the expectation-maximization (EM) algorithm to the data to be classified. Experiments carried out on a multisource and multitemporal data set confirmed the effectiveness of the proposed approach.

Neurodynamics of learning and network performance

A simple dynamic model of a neural network is presented. Using the dynamic model of a neural network, we improve the performance of a three-layer multilayer perceptron (MLP). The dynamic model of a MLP is used to make fundamental changes in the network optimization strategy. These changes are: Neuron activation functions are used, which reduce the probability of singular Jacobians; Successive regularization is used to constrain the volume of the weight space being minimized; Boltzmann pruning is used to constrain the dimension of the weight space; and prior class probabilities are used to normalize all error calculations, so that statistically significant samples of rare but important classes can be included without distortion of the error surface. All four of these changes are made in the inner loop of a conjugate gradient optimization iteration and are intended to simplify the training dynamics ofthe optimization. On handprinted digits and fingerprint classification problems, these modifications improve error-reject performance by factors between 2 and 4 and reduce network size by 40 to 60%.

Training Dynamics and Neural Network Performance

We use an analysis of a simple model of recurrent network dynamics to gain qualitative insights into the training dynamics of feedforward multilayer perceptrons (MLPs) used for classification. These insights suggest changes to the training methods used for MLPs that improve network performance significantly. In previous work, the probabilistic neural network (PNN) was shown to provide better zero-reject error performance on character and fingerprint classification problems than radial basis function and MLP-based neural network methods. We will show that performance equal to or better than PNN can be achieved with a single three-layer MLP by making fundamental changes in the network optimization strategy. These changes are: 1) use of neuron activation functions, which reduce the probability of singular Jacobians; 2) use of successive regularization to constrain the volume of the minimized weight space; 3) use of Boltzmann pruning to constrain the dimension of the weight space; 4) use of Prior class probabilities to normalize all error calculations, so that statistically significant samples of rare but important classes can be included without distorting the error surface. All four of these changes are made in the inner loop of a conjugate gradient optimization iteration and are intended to simplify the training dynamics of the optimization. On handprinted digits and fingerprint classification problems these modifications improve error-reject performance by factors between 2 and 4, and reduce network size by 40–60%. © 1997 Elsevier Science Ltd.