Posterior Class Probabilities Research Articles

Fine-resolution land cover mapping over large and mountainous areas for Lāna‘i, Hawaii using posterior probabilities, and expert knowledge

ABSTRACT The task of accurately mapping species-specific vegetation cover in remote and topographically complex regions like those found in Hawaiʻi presents unique challenges. This study leverages a machine learning approach to accurately classify vegetation into fine species-specific classes across the island of Lāna‘i, Hawaii, offering a novel methodology for tackling such challenges. Utilizing high-resolution WordView-2 satellite imagery, a neural network classifier and a custom lidar-based geometric correction, we introduced two new approaches to refine our high-resolution land cover classifications. This included the implementation of prior-based adjustments to class posterior probabilities to enhance land cover classification accuracy. Moreover, we developed mixed hierarchical classification maps that use class posterior probabilities to identify, at the pixel level, the finest land cover class that meets a user-defined confidence threshold. The resulting high-resolution land cover map for Lāna‘i captures the rich diversity and distribution of native and invasive plant species with high overall accuracy, generally exceeding 95%, based on independent ground control data. The capacity to produce wall-to-wall species-level vegetation maps provides a new window into monitoring vegetation dynamics on Lāna‘i and similarly remote and topographically complex regions, and contributes to our broader understanding of ecosystem responses to invasive species, climatic changes, and land management practices such as erosion and sediment control planning. Our approach offers a blueprint for similar efforts in other complex and remote ecosystems.

RNA-Seq-Based Molecular Classification Analyses in Colorectal Cancer and Synchronous Adenoma.

Colorectal cancers (CRC) are classified into consensus molecular subtypes (CMS) based on gene expression profiles. The revised classification system iCMS was proposed by considering intrinsic epithelial status, microsatellite instability (MSI), and fibrosis. This study aimed to provide molecular evidence for the adenoma-carcinoma sequence concept by examining CRC and synchronous adenomas using iCMS. Epithelial CMS cell proportion was estimated using CiberSortx, an in silico cell fractionation method that included CMS cell types among the reference cell types. A random forest (RF) model estimated the posterior probabilities of CMS classes, which were compared with the CiberSortx results. Gene expression profiles of the published iCMS signature panel were retrieved from our dataset and subjected to heatmap clustering for classification. Bulk RNA sequencing data were collected from 29 adenocarcinomas and 11 adenoma samples. CiberSortx showed all CRC contained either CMS2 or CMS3 as the major epithelial cancer cell type. The RF model classified approximately half of the CRC as CMS4, whereas CMS4 was hardly detected by CiberSortx. Because they were enriched with myofibroblasts as per the CiberSortx classification, we tentatively designated them as iCMS2-F/iCMS3-F. iCMS coupled with the application of an in silico cell fractionation method can provide the molecular dissection of CRC and adenoma.

Epistemic uncertainty aware semantic localization and mapping for inference and belief space planning

We investigate the problem of autonomous object classification and semantic SLAM, which generally exhibits a tight coupling between classification, metric SLAM, and planning under uncertainty. We contribute a unified framework for inference and belief space planning (BSP) that addresses prominent sources of uncertainty in this context: classification aliasing (classifier cannot distinguish between candidate classes from certain viewpoints), classifier epistemic uncertainty (classifier receives data “far” from its training set), and localization uncertainty (camera and object poses are uncertain). Specifically, two methods are developed for maintaining a joint distribution over robot and object poses, and over a posterior class probability vector that considers epistemic uncertainty in a Bayesian fashion. The first approach is Multi-Hybrid (MH), where multiple hybrid beliefs over poses and classes are maintained to approximate the joint belief over poses and posterior class probability. The second approach is Joint Lambda Pose (JLP), where the joint belief is maintained directly using a novel JLP factor. Furthermore, we extend both methods to BSP, while reasoning about future posterior epistemic uncertainty indirectly or directly via a novel information-theoretic reward function. Both inference methods utilize a novel viewpoint-dependent classifier uncertainty model that leverages the coupling between poses and classification scores and predicts the epistemic uncertainty from certain viewpoints. In addition, this model is used to generate predicted measurements during planning. To the best of our knowledge, this is the first work that reasons about classifier epistemic uncertainty within semantic SLAM and BSP. We extensively evaluate our inference and BSP approaches in simulation using real image data from the Active Vision Dataset. Results clearly indicate superior classification performance of our methods compared to an approach that is not epistemic uncertainty aware.

Accounting for careless and insufficient effort responding in large-scale survey data—development, evaluation, and application of a screen-time-based weighting procedure

Careless and insufficient effort responding (C/IER) poses a major threat to the quality of large-scale survey data. Traditional indicator-based procedures for its detection are limited in that they are only sensitive to specific types of C/IER behavior, such as straight lining or rapid responding, rely on arbitrary threshold settings, and do not allow taking the uncertainty of C/IER classification into account. Overcoming these limitations, we develop a two-step screen-time-based weighting procedure for computer-administered surveys. The procedure allows considering the uncertainty in C/IER identification, is agnostic towards the specific types of C/IE response patterns, and can feasibly be integrated with common analysis workflows for large-scale survey data. In Step 1, we draw on mixture modeling to identify subcomponents of log screen time distributions presumably stemming from C/IER. In Step 2, the analysis model of choice is applied to item response data, with respondents’ posterior class probabilities being employed to downweigh response patterns according to their probability of stemming from C/IER. We illustrate the approach on a sample of more than 400,000 respondents being administered 48 scales of the PISA 2018 background questionnaire. We gather supporting validity evidence by investigating relationships between C/IER proportions and screen characteristics that entail higher cognitive burden, such as screen position and text length, relating identified C/IER proportions to other indicators of C/IER as well as by investigating rank-order consistency in C/IER behavior across screens. Finally, in a re-analysis of the PISA 2018 background questionnaire data, we investigate the impact of the C/IER adjustments on country-level comparisons.

Moving towards a taxonomy of cognitive impairments in epilepsy: application of latent profile analysis to 1178 patients with temporal lobe epilepsy.

In efforts to understand the cognitive heterogeneity within and across epilepsy syndromes, cognitive phenotyping has been proposed as a new taxonomy aimed at developing a harmonized approach to cognitive classification in epilepsy. Data- and clinically driven approaches have been previously used with variability in the phenotypes derived across studies. In our study, we utilize latent profile analysis to test several models of phenotypes in a large multicentre sample of patients with temporal lobe epilepsy and evaluate their demographic and clinical profiles. For the first time, we examine the added value of replacing missing data and examine factors that may be contributing to missingness. A sample of 1178 participants met the inclusion criteria for the study, which included a diagnosis of temporal lobe epilepsy and the availability of comprehensive neuropsychological data. Models with two to five classes were examined using latent profile analysis and the optimal model was selected based on fit indices, posterior probabilities and proportion of sample sizes. The models were also examined with imputed data to investigate the impact of missing data on model selection. Based on the fit indices, posterior probability and distinctiveness of the latent classes, a three-class solution was the optimal solution. This three-class solution comprised a group of patients with multidomain impairments, a group with impairments predominantly in language and a group with no impairments. Overall, the multidomain group demonstrated a worse clinical profile and comprised a greater proportion of patients with mesial temporal sclerosis, a longer disease duration and a higher number of anti-seizure medications. The four-class and five-class solutions demonstrated the lowest probabilities of a group membership. Analyses with imputed data demonstrated that the four-class solution was the optimal solution; however, there was a weak agreement between the missing and imputed data sets for the four-Class solutions (κ = 0.288, P < 0.001). This study represents the first to use latent profile analysis to test and compare multiple models of cognitive phenotypes in temporal lobe epilepsy and to determine the impact of missing data on model fit. We found that the three-phenotype model was the most meaningful based on several fit indices and produced phenotypes with unique demographic and clinical profiles. Our findings demonstrate that latent profile analysis is a rigorous method to identify phenotypes in large, heterogeneous epilepsy samples. Furthermore, this study highlights the importance of examining the impact of missing data in phenotyping methods. Our latent profile analysis-derived phenotypes can inform future studies aimed at identifying cognitive phenotypes in other neurological disorders.

Intelligent fault diagnosis of bevel gearboxes using semi-supervised probability support matrix machine and infrared imaging

Fault diagnosis is of great significance to ensure the reliability and safety of complex bevel gearbox systems. Most existing intelligent fault diagnosis approaches of bevel gearboxes are designed with vibration monitoring. However, the collected vibration data are vulnerable to noise pollution and machinery operating conditions. Besides, traditional fault diagnosis models highly rely on numerous labeled samples, and neglect the high cost of label annotation in real-world applications. Therefore, a novel fault diagnosis approach based on semi-supervised probability support matrix machine (SPSMM) and infrared imaging is proposed for bevel gearboxes in this paper, which has the following properties. Firstly, SPSMM classifies 2D matrix data directly without vectorization, thus fully utilizing the spatial information in infrared images. Secondly, a probability output strategy is designed for SPSMM to calculate the posterior class probability estimation of matrix inputs, and consequently enhance the diagnostic accuracy and robustness of the model. Thirdly, a semi-supervised learning (SSL) framework is proposed for SPSMM to carry out sample transfer from the unlabeled sample pool to the labeled sample pool, which can effectively alleviate the problem of insufficient labeled samples. The superiority of the proposed diagnosis approach is demonstrated with an infrared imaging dataset of a bevel gearbox.

Cautious weighted random forests

Random forest is an efficient and accurate classification model, which makes decisions by aggregating a set of trees, either by voting or by averaging class posterior probability estimates. However, tree outputs may be unreliable in presence of scarce data. The imprecise Dirichlet model (IDM) provides workaround, by replacing point probability estimates with interval-valued ones. This paper investigates a new tree aggregation method based on the theory of belief functions to combine such probability intervals, resulting in a cautious random forest classifier. In particular, we propose a strategy for computing tree weights based on the minimization of a convex cost function, which takes both determinacy and accuracy into account and makes it possible to adjust the level of cautiousness of the model. The proposed model is evaluated on 25 UCI datasets and is demonstrated to be more adaptive to the noise in training data and to achieve a better compromise between informativeness and cautiousness.

Factorizable Joint Shift in Multinomial Classification

Factorizable joint shift (FJS) was recently proposed as a type of dataset shift for which the complete characteristics can be estimated from feature data observations on the test dataset by a method called Joint Importance Aligning. For the multinomial (multiclass) classification setting, we derive a representation of factorizable joint shift in terms of the source (training) distribution, the target (test) prior class probabilities and the target marginal distribution of the features. On the basis of this result, we propose alternatives to joint importance aligning and, at the same time, point out that factorizable joint shift is not fully identifiable if no class label information on the test dataset is available and no additional assumptions are made. Other results of the paper include correction formulae for the posterior class probabilities both under general dataset shift and factorizable joint shift. In addition, we investigate the consequences of assuming factorizable joint shift for the bias caused by sample selection.

Bayesian Distance Weighted Discrimination

Distance weighted discrimination (DWD) is a linear discrimination method that is particularly well-suited for classification tasks with high-dimensional data. The DWD coefficients minimize an intuitive objective function, which can solved efficiently using state-of-the-art optimization techniques. However, DWD has not yet been cast into a model-based framework for statistical inference. In this article we show that DWD identifies the mode of a proper Bayesian posterior distribution, that results from a particular link function for the class probabilities and a shrinkage-inducing proper prior distribution on the coefficients. We describe a relatively efficient Markov chain Monte Carlo (MCMC) algorithm to simulate from the true posterior under this Bayesian framework. We show that the posterior is asymptotically normal and derive the mean and covariance matrix of its limiting distribution. Through several simulation studies and an application to breast cancer genomics we demonstrate how the Bayesian approach to DWD can be used to (a) compute well-calibrated posterior class probabilities, (b) assess uncertainty in the DWD coefficients and resulting sample scores, (c) improve power via semisupervised analysis when not all class labels are available, and (d) automatically determine a penalty tuning parameter within the model-based framework. R code to perform Bayesian DWD is available at https://github.com/lockEF/BayesianDWD. Supplementary materials for this article are available online.

Gait-Based Person Identification and Intruder Detection Using mm-Wave Sensing in Multi-Person Scenario

Using mm-wave sensing to identify persons by the way they walk has recently emerged as a promising solution of biometrics, and has found versatile applications in security surveillance, automatic access control, and health monitoring. In this paper, we build a system named MGait for indoor multi-person identification and intruder detection based on gait micro-Doppler (m-D) signatures captured by a low-cost mm-wave radar. In multi-person scenario where multiple subjects concurrently share and walk within the same physical space, the proposed system continuously detects and separately tracks each subject in the range-Doppler (R-D) space frame by frame to extract their respective gait m-D signatures. Then, an open-set identification network is trained by a large-margin Gaussian mixture (L-GM) loss to learn highly discriminative feature representations and compel the learned features of the training set to follow a Gaussian mixture distribution, with each component representing a registered user. As such, the known users can be directly identified based on the class-posterior probability and the intruder can also be rejected by setting a probability threshold. The proposed system is verified on real radar measurements collected by a 77 GHz FMCW radar, achieving an overall accuracy of 88.59&#x0025; in identifying up to 5 subjects, including 1 known user and 4 intruders, freely and concurrently moving in a corridor space.

Land cover change detection in the Aralkum with multi-source satellite datasets

ABSTRACT The Aral Sea, once the fourth largest freshwater lake on Earth, has lost circa 90% of its original water surface in 1960. Maps of different land cover categories provide a suitable baseline to plan and implement effective measures to combat ongoing desertification, such as reforestation of dried out Aral Sea soils. In this study, we used satellite-based remote sensing data and applied a machine learning method (Random Forest) to map land cover in the Aralkum in 2020. We tested different satellite data from optical (Landsat-8, Sentinel-2) and Radar instruments (Sentinel-1) and trained a random forest model for classifying different combinations of these data sets into ten distinct land cover classes. We further calculated per-pixel uncertainty based on posterior classification probability scores. An accuracy assessment, based on in-situ data, revealed that the average overall accuracy of land cover maps was 86.8%. Fusing optical and radar instruments achieved the highest overall accuracy (88.8%, with lower/higher 95% confidence interval values of 87.6%/89.9%, and a Kappa value of 0.865. Classification uncertainty was lower in more homogeneous landscapes (i.e. large expanses of a single land cover class like water or shrubland). Only around 9% of the study area was still water in 2020, while 32% was covered by saline soils with high erosion risk. Several potential applications of this land cover map in the Aralkum exist – spanning many areas of environmental impact assessment, policy, and planning and management or afforestation. This methodological framework can similarly provide a useful template for more broadly assessing large-scale, land dynamics at high-resolution in the entire Aralkum and surrounding areas.

CHAOS in the Home Environment and Child Weight-Related Outcomes

Biopsychosocial approaches to health care are critical to addressing childhood obesity. This study aimed to examine how multiple indicators of the home environment related to child weight-related outcomes. We hypothesized that families with home environments of higher chaos and stress, and lower quality parent-child interactions, would have children with a higher body mass index (BMI), less healthy dietary intake, and less healthy eating behaviors. Data were drawn from the cross-sectional Phase I of the Family Matters study. Participants were 150 racially/ethnically diverse families with a child between 5 to 7 (mean, 6.4) years old. We used a latent profile analysis approach. A 4-class solution fit the data well, and we used predicted class posterior probabilities to assign families to classes. We then regressed the results onto the distal outcomes of child BMI, healthy dietary intake, and healthy eating behaviors. Families were classified as Collaborative-Chill (n = 38), Busy Bees (n = 37), Engaged (n = 61), and Inconsistent-Distant (n = 14). Collaborative-Chill was used as the reference class. Inconsistent-Distant families had children with higher BMI (P < .001) that were more food responsive (P < .001). Busy Bees families had children who were more food responsive (P = .04) and more satiety responsive (P = .02). Engaged families had children who were marginally more food responsive (P = .06). Household chaos, parent stress, and parent-child interactions are important components of the home environment implicated in children's weight-related outcomes. Health care providers should consider these indicators with child patients who struggle with obesity.

A Novel Hybrid Learning System Using Modified Breaking Ties Algorithm and Multinomial Logistic Regression for Classification and Segmentation of Hyperspectral Images

A new methodology, the hybrid learning system (HLS), based upon semi-supervised learning is proposed. HLS categorizes hyperspectral images into segmented regions with discriminative features using reduced training size. The technique utilizes the modified breaking ties (MBT) algorithm for active learning and unsupervised learning-based regressors, viz. multinomial logistic regression, for hyperspectral image categorization. The probabilities estimated by multinomial logistic regression for each sample helps towards improved segregation. The high dimensionality leads to a curse of dimensionality, which ultimately deteriorates the performance of remote sensing data classification, and the problem aggravates further if labeled training samples are limited. Many studies have tried to address the problem and have employed different methodologies for remote sensing data classification, such as kernelized methods, because of insensitiveness towards the utilization of large dataset information and active learning (AL) approaches (breaking ties as a representative) to choose only prominent samples for training data. The HLS methodology proposed in the current study is a combination of supervised and unsupervised training with generalized composite kernels generating posterior class probabilities for classification. In order to retrieve the best segmentation labels, we employed Markov random fields, which make use of prior labels from the output of the multinomial logistic regression. The comparison of HLS was carried out with known methodologies, using benchmark hyperspectral imaging (HI) datasets, namely “Indian Pines” and “Pavia University”. Findings of this study show that the HLS yields the overall accuracy of {99.93% and 99.98%}Indian Pines and {99.14% and 99.42%}Pavia University for classification and segmentation, respectively.

An Improved Boundary Uncertainty-Based Estimation for Classifier Evaluation

This paper proposes a new boundary uncertainty-based estimation method that has significantly higher accuracy, scalability, and applicability than our previously proposed boundary uncertainty estimation method. In our previous work, we introduced a new classifier evaluation metric that we termed “boundary uncertainty.” The name “boundary uncertainty” comes from evaluating the classifier based solely on measuring the equality between class posterior probabilities along the classifier boundary; satisfaction of such equality can be described as “uncertainty” along the classifier boundary. We also introduced a method to estimate this new evaluation metric. By focusing solely on the classifier boundary to evaluate its uncertainty, boundary uncertainty defines an easier estimation target that can be accurately estimated based directly on a finite training set without using a validation set. Regardless of the dataset, boundary uncertainty is defined between 0 and 1, where 1 indicates whether probability estimation for the Bayes error is achieved. We call our previous boundary uncertainty estimation method “Proposal 1” in order to contrast it with the new method introduced in this paper, which we call “Proposal 2.” Using Proposal 1, we performed successful classifier evaluation on real-world data and supported it with theoretical analysis. However, Proposal 1 suffered from accuracy, scalability, and applicability limitations owing to the difficulty of finding the location of a classifier boundary in a multidimensional sample space. The novelty of Proposal 2 is that it locally reformalizes boundary uncertainty in a single dimension that focuses on the classifier boundary. This convenient reduction with a focus toward the classifier boundary provides the new method’s significant improvements. In classifier evaluation experiments on Support Vector Machines (SVM) and MultiLayer Perceptron (MLP), we demonstrate that Proposal 2 offers a competitive classifier evaluation accuracy compared to a benchmark Cross Validation (CV) method as well as much higher scalability than both CV and Proposal 1.

Softmax Model as Generalization upon Logistic Discrimination Suffers from Overfitting

The motivation behind this paper is to investigate the use of Softmax model for classification. We show that Softmax model is a nonlinear generalization for the logistic discrimination, that can approximate the posterior probabilities of classes where other Artificial neural network (ANN) models don't have this ability. We show that Softmax model has more flexibility than logistic discrimination in terms of correct classification. To show the performance of Softmax model a medical data set on thyroid gland state is used. The result is that Softmax model may suffer from overfitting.

Characterization and classification of Romanian acacia honey based on its physicochemical parameters and chemometrics

Three groups of Romanian acacia honey, i.e., pure, directly adulterated (by mixing the pure honey with three sugar syrups), and indirectly adulterated (by feeding the bees with the same syrups), were characterized and discriminated based on their physicochemical parameters. Moisture, ash, 5-hydroxymethylfurfural (HMF), reducing sugars (fructose and glucose), and sucrose contents, free acidity, diastase activity, ratio between stable carbon isotopes of honey and its proteins (δ13CH and δ13CP) were evaluated. Adulteration led to a significant increase in sucrose content, HMF level, and Δδ13C = δ13CH‒δ13CP as well a decrease in reducing sugar content and diastase activity. Principal component analysis (PCA) and linear discriminant analysis (LDA) were applied to experimental data in order to distinguish between pure and adulterated honey. The most relevant discriminative parameters were diastase activity, HMF, sucrose, and reducing sugar contents. Posterior classification probabilities and classification functions obtained by LDA revealed that 100% of honey samples were correctly assigned to their original group.

Land-Use/Land-Cover Change Detection Based on Class-Prior Object-Oriented Conditional Random Field Framework for High Spatial Resolution Remote Sensing Imagery

High spatial resolution (HSR) remote sensing images can reflect more subtle changes and more specific types of land use and land cover (LULC) due to the abundant spatial geometric information. In this article, a class-prior object-oriented conditional random field (COCRF) framework consisting of a binary change detection (CD) task and a multiclass CD task is proposed to fill the application gap. In the proposed framework, the class-prior knowledge is used to improve the construction of the unary potential in both the binary and multiclass CD tasks, to reduce the influence of spectral variability. The binary CD result provides a constraint to the multiclass CD result. As a result, both parts have effective interaction. The class posterior probability images of two dates can be obtained automatically with the class-prior knowledge by sample migration. Furthermore, an object constraint described by the class dispersion within the objects is added to improve the smoothness in local objects, while the pairwise potential improves the smoothness of the whole area by using the eight-neighborhood spectral information of the center pixel. By integrating the above approaches, the problems of error accumulation and the manual intervention required in the traditional multiclass CD methods can be relieved. An adaptive parameter estimation strategy is also adopted in the proposed framework, to save the time required for manual parameter setting. The proposed COCRF framework was validated on two HSR remote sensing image data sets, where it achieved a better performance than the other state-of-the-art CD methods.

Semantic Fisher Scores for Task Transfer: Using Objects to Classify Scenes.

The transfer of a neural network (CNN) trained to recognize objects to the task of scene classification is considered. A Bag-of-Semantics (BoS) representation is first induced, by feeding scene image patches to the object CNN, and representing the scene image by the ensuing bag of posterior class probability vectors (semantic posteriors). The encoding of the BoS with a Fisher vector (FV) is then studied. A link is established between the FV of any probabilistic model and the Q-function of the expectation-maximization (EM) algorithm used to estimate its parameters by maximum likelihood. This enables 1) immediate derivation of FVs for any model for which an EM algorithm exists, and 2) leveraging efficient implementations from the EM literature for the computation of FVs. It is then shown that standard FVs, such as those derived from Gaussian or even Dirichlet mixtures, are unsuccessful for the transfer of semantic posteriors, due to the highly non-linear nature of the probability simplex. The analysis of these FVs shows that significant benefits can ensue by 1) designing FVs in the natural parameter space of the multinomial distribution, and 2) adopting sophisticated probabilistic models of semantic feature covariance. The combination of these two insights leads to the encoding of the BoS in the natural parameter space of the multinomial, using a vector of Fisher scores derived from a mixture of factor analyzers (MFA). A network implementation of the MFA Fisher Score (MFA-FS), denoted as the MFAFSNet, is finally proposed to enable end-to-end training. Experiments with various object CNNs and datasets show that the approach has state-of-the-art transfer performance. Somewhat surprisingly, the scene classification results are superior to those of a CNN explicitly trained for scene classification, using a large scene dataset (Places). This suggests that holistic analysis is insufficient for scene classification. The modeling of local object semantics appears to be at least equally important. The two approaches are also shown to be strongly complementary, leading to very large scene classification gains when combined, and outperforming all previous scene classification approaches by a sizable margin.

Deep Neural Network Self-Distillation Exploiting Data Representation Invariance.

To harvest small networks with high accuracies, most existing methods mainly utilize compression techniques such as low-rank decomposition and pruning to compress a trained large model into a small network or transfer knowledge from a powerful large model (teacher) to a small network (student). Despite their success in generating small models of high performance, the dependence of accompanying assistive models complicates the training process and increases memory and time cost. In this article, we propose an elegant self-distillation (SD) mechanism to obtain high-accuracy models directly without going through an assistive model. Inspired by the invariant recognition in the human vision system, different distorted instances of the same input should possess similar high-level data representations. Thus, we can learn data representation invariance between different distorted versions of the same sample. Especially, in our learning algorithm based on SD, the single network utilizes the maximum mean discrepancy metric to learn the global feature consistency and the Kullback-Leibler divergence to constrain the posterior class probability consistency across the different distorted branches. Extensive experiments on MNIST, CIFAR-10/100, and ImageNet data sets demonstrate that the proposed method can effectively reduce the generalization error for various network architectures, such as AlexNet, VGGNet, ResNet, Wide ResNet, and DenseNet, and outperform existing model distillation methods with little extra training efforts.

On the appropriateness of Platt scaling in classifier calibration

Many applications using data mining and machine learning techniques require posterior probability estimates besides often highly accurate predictions. Classifier calibration is a separate branch of machine learning that aims at transforming classifier predictions into posterior class probabilities and thus are useful additional extensions in the respective applications. Among the existing state-of-the-art classifier calibration techniques, Platt scaling (sometimes also called sigmoid or logistic calibration) actually is the only parametric one while almost all of its competing methods do not rely on parametric assumptions. Platt scaling is controversially discussed in the classifier calibration literature, despite good empirical results reported in many domains there are many authors criticizing it. Interestingly, none of these criticisms properly deal with the underlying parametric assumptions. Instead, even incorrect statements exist. Thus, the first contribution of this work is to review such criticism and to present a proof of the true parametric assumptions. In fact, these are more general and valid for different probability distributions, which is an immediate consequence. Next, the relationship between Platt scaling and a different, relatively new classifier calibration technique called beta calibration is analyzed and it is shown that these two are actually equivalent: Their only difference lies in the characteristics of the classifier whose predictions are calibrated. Thus, the proven validity of Platt scaling additionally translates directly into a proven optimality of beta calibration. Furthermore, evaluating classifier calibration techniques is a highly non-trivial problem as the true posteriors cannot be used as a reference. Hence, the existing evaluation metrics are reviewed as well because there exist relatively popular evaluation criteria that should not be used at all. Finally, the theoretical findings are supported by a simulation study.