Prior Class Probabilities Research Articles

A Comparative Study of Landslide Susceptibility Mapping Using Bagging PU Learning in Class-Prior Probability Shift Datasets

Landslide susceptibility mapping is typically based on binary prediction probabilities. However, non-landslide samples in modeling datasets are often unlabeled data, and the phenomenon of class-priori shift, that is, the proportion of landslide samples frequently deviates from real-world scenarios and is spatially heterogeneous. By comparing the classification performance and predicted probability distributions across multiple unbalanced datasets with known and unknown sample proportions, this study assesses the landslide susceptibility model’s generalization ability in the context of class-prior shifts. The study investigates the potential of Bagging PU Learning, a semi-supervised learning approach, in improving the generalization performance of landslide susceptibility models and proposes the Bagging PU-GDBT algorithm. Our findings highlight the effectiveness of Bagging PU Learning in enhancing the recall of landslides and the generalization capabilities of models on unbalanced datasets. This method reduces prediction uncertainties, especially in high and very high susceptibility zones. Furthermore, results emphasize the superiority of models trained on balanced datasets with 1:1 sample ratio for landslide susceptibility mapping over those trained on unbalanced datasets.

Expand and Shrink: Federated Learning with Unlabeled Data Using Clustering

The amalgamation of the Internet of Things (IoT) and federated learning (FL) is leading the next generation of data usage due to the possibility of deep learning with data privacy preservation. The FL architecture currently assumes labeled data samples from a client for supervised classification, which is unrealistic. Most research works in the literature focus on local training, update receiving, and global model updates. However, by principle, the labeling must be performed on the client side because the data samples cannot leave the source under the FL principle. In the literature, a few works have proposed methods for unlabeled data for FL using “class-prior probabilities” or “pseudo-labeling”. However, these methods make either unrealistic or uncommon assumptions, such as knowing class-prior probabilities are impractical or unavailable for each classification task and even more challenging in the IoT ecosystem. Considering these limitations, we explored the possibility of performing federated learning with unlabeled data by providing a clustering-based method of labeling the sample before training or federation. The proposed work will be suitable for every type of classification task. We performed different experiments on the client by varying the labeled data ratio, the number of clusters, and the client participation ratio. We achieved accuracy rates of 87% and 90% by using 0.01 and 0.03 of the truth labels, respectively.

Whole slide image representation in bone marrow cytology

One of the goals of AI-based computational pathology is to generate compact representations of whole slide images (WSIs) that capture the essential information needed for diagnosis. While such approaches have been applied to histopathology, few applications have been reported in cytology. Bone marrow aspirate cytology is the basis for key clinical decisions in hematology. However, visual inspection of aspirate specimens is a tedious and complex process subject to variation in interpretation, and hematopathology expertise is scarce. The ability to generate a compact representation of an aspirate specimen may form the basis for clinical decision-support tools in hematology. In this study, we leverage our previously published end-to-end AI-based system for counting and classifying cells from bone marrow aspirate WSIs, which enables the direct use of individual cells as inputs rather than WSI patches. We then construct bags of individual cell features from each WSI, and apply multiple instance learning to extract their vector representations. To evaluate the quality of our representations, we conducted WSI retrieval and classification tasks. Our results show that we achieved a mAP@10 of 0.58 ±0.02 in WSI-level image retrieval, surpassing the random-retrieval baseline of 0.39 ±0.1. Furthermore, we predicted five diagnostic labels for individual aspirate WSIs with a weighted-average F1 score of 0.57 ±0.03 using a k-nearest-neighbors (k-NN) model, outperforming guessing using empirical class prior probabilities (0.26 ±0.02). We present the first example of exploring trainable mechanisms to generate compact, slide-level representations in bone marrow cytology with deep learning. This method has the potential to summarize complex semantic information in WSIs toward improved diagnostics in hematology, and may eventually support AI-assisted computational pathology approaches.

Threshold optimization and random undersampling for imbalanced credit card data

Output thresholding is well-suited for addressing class imbalance, since the technique does not increase dataset size, run the risk of discarding important instances, or modify an existing learner. Through the use of the Credit Card Fraud Detection Dataset, this study proposes a threshold optimization approach that factors in the constraint True Positive Rate (TPR) ≥ True Negative Rate (TNR). Our findings indicate that an increase of the Area Under the Precision–Recall Curve (AUPRC) score is associated with an improvement in threshold-based classification scores, while an increase of positive class prior probability causes optimal thresholds to increase. In addition, we discovered that best overall results for the selection of an optimal threshold are obtained without the use of Random Undersampling (RUS). Furthermore, with the exception of AUPRC, we established that the default threshold yields good performance scores at a balanced class ratio. Our evaluation of four threshold optimization techniques, eight threshold-dependent metrics, and two threshold-agnostic metrics defines the uniqueness of this research.

Factorizable Joint Shift in Multinomial Classification

Factorizable joint shift (FJS) was recently proposed as a type of dataset shift for which the complete characteristics can be estimated from feature data observations on the test dataset by a method called Joint Importance Aligning. For the multinomial (multiclass) classification setting, we derive a representation of factorizable joint shift in terms of the source (training) distribution, the target (test) prior class probabilities and the target marginal distribution of the features. On the basis of this result, we propose alternatives to joint importance aligning and, at the same time, point out that factorizable joint shift is not fully identifiable if no class label information on the test dataset is available and no additional assumptions are made. Other results of the paper include correction formulae for the posterior class probabilities both under general dataset shift and factorizable joint shift. In addition, we investigate the consequences of assuming factorizable joint shift for the bias caused by sample selection.

Multi-instance positive and unlabeled learning with bi-level embedding

Multiple Instance Learning (MIL) is a widely studied learning paradigm which arises from real applications. Existing MIL methods have achieved prominent performances under the premise of plenty annotation data. Nevertheless, sufficient labeled data is often unattainable due to the high labeling cost. For example, the task in web image identification is to find similar samples among a large size of unlabeled dataset through a small number of provided target pictures. This leads to a particular scenario of Multiple Instance Learning with insufficient Positive and superabundant Unlabeled data (PU-MIL), which is a hot research topic in MIL recently. In this paper, we propose a novel method called Multiple Instance Learning with Bi-level Embedding (MILBLE) to tackle PU-MIL problem. Unlike other PU-MIL method using only simple single-level mapping, the bi-level embedding strategy are designed to customize specific mapping for positive and unlabeled data. It ensures the characteristics of key instance are not erased. Moreover, the weighting measure adopted in positive data can extracts the uncontaminated information of true positive instances without interference from negative ones. Finally, we minimize the classification error loss of mapped examples based on class-prior probability to train the optimal classifier. Experimental results show that our method has better performance than other state-of-the-art methods.

From Human Explanation to Model Interpretability: A Framework Based on Weight of Evidence

We take inspiration from the study of human explanation to inform the design and evaluation of interpretability methods in machine learning. First, we survey the literature on human explanation in philosophy, cognitive science, and the social sciences, and propose a list of design principles for machine-generated explanations that are meaningful to humans. Using the concept of weight of evidence from information theory, we develop a method for generating explanations that adhere to these principles. We show that this method can be adapted to handle high-dimensional, multi-class settings, yielding a flexible framework for generating explanations. We demonstrate that these explanations can be estimated accurately from finite samples and are robust to small perturbations of the inputs. We also evaluate our method through a qualitative user study with machine learning practitioners, where we observe that the resulting explanations are usable despite some participants struggling with background concepts like prior class probabilities. Finally, we conclude by surfacing design implications for interpretability tools in general.

Estimating the class prior for positive and unlabelled data via logistic regression

In the paper, we revisit the problem of class prior probability estimation with positive and unlabelled data gathered in a single-sample scenario. The task is important as it is known that in positive unlabelled setting, a classifier can be successfully learned if the class prior is available. We show that without additional assumptions, class prior probability is not identifiable and thus the existing non-parametric estimators are necessarily biased in general if extra assumptions are not imposed. The magnitude of their bias is also investigated. The problem becomes identifiable when the probabilistic structure satisfies mild semi-parametric assumptions. Consequently, we propose a method based on a logistic fit and a concave minorization of its (non-concave) log-likelihood. The experiments conducted on artificial and benchmark datasets as well as on a large clinical database MIMIC indicate that the estimation errors for the proposed method are usually lower than for its competitors and that it is robust against departures from logistic settings.

A Critical Reassessment of the Saerens-Latinne-Decaestecker Algorithm for Posterior Probability Adjustment

We critically re-examine the Saerens-Latinne-Decaestecker (SLD) algorithm, a well-known method for estimating class prior probabilities (“priors”) and adjusting posterior probabilities (“posteriors”) in scenarios characterized by distribution shift, i.e., difference in the distribution of the priors between the training and the unlabelled documents. Given a machine learned classifier and a set of unlabelled documents for which the classifier has returned posterior probabilities and estimates of the prior probabilities, SLD updates them both in an iterative, mutually recursive way, with the goal of making both more accurate; this is of key importance in downstream tasks such as single-label multiclass classification and cost-sensitive text classification. Since its publication, SLD has become the standard algorithm for improving the quality of the posteriors in the presence of distribution shift, and SLD is still considered a top contender when we need to estimate the priors (a task that has become known as “quantification”). However, its real effectiveness in improving the quality of the posteriors has been questioned. We here present the results of systematic experiments conducted on a large, publicly available dataset, across multiple amounts of distribution shift and multiple learners. Our experiments show that SLD improves the quality of the posterior probabilities and of the estimates of the prior probabilities, but only when the number of classes in the classification scheme is very small and the classifier is calibrated. As the number of classes grows, or as we use non-calibrated classifiers, SLD converges more slowly (and often does not converge at all), performance degrades rapidly, and the impact of SLD on the quality of the prior estimates and of the posteriors becomes negative rather than positive.

Information-Theoretic Representation Learning for Positive-Unlabeled Classification.

Recent advances in weakly supervised classification allow us to train a classifier from only positive and unlabeled (PU) data. However, existing PU classification methods typically require an accurate estimate of the class-prior probability, a critical bottleneck particularly for high-dimensional data. This problem has been commonly addressed by applying principal component analysis in advance, but such unsupervised dimension reduction can collapse the underlying class structure. In this letter, we propose a novel representation learning method from PU data based on the information-maximization principle. Our method does not require class-prior estimation and thus can be used as a preprocessing method for PU classification. Through experiments, we demonstrate that our method, combined with deep neural networks, highly improves the accuracy of PU class-prior estimation, leading to state-of-the-art PU classification performance.

Incorporating spatial association into statistical classifiers: local pattern-based prior tuning

ABSTRACT This paper proposes a new classification method for spatial data by adjusting prior class probabilities according to local spatial patterns. First, the proposed method uses a classical statistical classifier to model training data. Second, the prior class probabilities are estimated according to the local spatial pattern and the classifier for each unseen object is adapted using the estimated prior probability. Finally, each unseen object is classified using its adapted classifier. Because the new method can be coupled with both generative and discriminant statistical classifiers, it performs generally more accurately than other methods for a variety of different spatial datasets. Experimental results show that this method has a lower prediction error than statistical classifiers that take no spatial information into account. Moreover, in the experiments, the new method also outperforms spatial auto-logistic regression and Markov random field-based methods when an appropriate estimate of local prior class distribution is used.

Cost-sensitive support vector machines

Many machine learning applications involve imbalance class prior probabilities, multi-class classification with many classes (often addressed by one-versus-rest strategy), or “cost-sensitive” classification. In such domains, each class (or in some cases, each sample) requires special treatment. In this paper, we use a constructive procedure to extend SVM’s standard loss function to optimize the classifier with respect to class imbalance or class costs. By drawing connections between risk minimization and probability elicitation, we show that the resulting classifier guarantees Bayes consistency. We further analyze the primal and the dual objective functions and derive the objective function in a regularized risk minimization framework. Finally, we extend the classifier to the with cost-sensitive learning with example dependent costs. We perform experimental analysis on class imbalance, cost-sensitive learning with given class and example costs and show that proposed algorithm provides superior generalization performance, compared to conventional methods.

A personalized mail re-filtering system based on the client

Email is an essential communication tool, but a large number of spam emails can seriously affect the work and life of users and can even cause property damage. Due to different interests and hobbies, there may be huge differences in the definition of spam by users; the realization of personalized spam filtering has become an important issue in the field of spam filtering. When emails are misjudged, the user has to manually modify it, which brings great inconvenience to the user experience. In order to effectively solve the above problems and realize the functions of personalized email filtering and automatic correction of mis-filtered emails, this paper combined with rules and statistical methods presents a personalized email re-filtering system based on the client (PRFC) and implements the automatic modification of the mis-filtered emails. A large part of existing spam filters do not consider the difference between class prior probability and class imbalance problem; they only filter the mail online. Firstly, the proposed filter system processes the mails entering the inbox and the garbage and then designs two mutually learned filters based on the multi-task learning principle to be used for the automatic modification of the mis-filtered emails in inbox and garbage. To ensure the performance of the filter based on the interests of users and data distribution of mails varying with time, a multi-window learning framework that combines important weights to effectively implement the dynamic adaptation of the filter was designed. Finally, our proposed filtering system on the TREC 2006c and 2007p data sets that gets a significant filtering efficiency was verified.

Image Classification by Integrating Reject Option and Prior Information

The accuracy in land-cover classification using remotely sensed imagery can be increased using Bayesian methods that incorporate prior probabilities of classes. However, estimating these prior probabilities can be expensive and data intensive. We propose methods to improve the classification accuracy using Bayesian methods to classify ambiguous (or low-confidence) pixels, using only the remotely sensed imagery or existing land-cover maps to estimate prior probabilities. We propose a spatial method that predicts prior probabilities from the original image, and a temporal method that incorporates land-cover maps from previous years. We illustrate our methods with a neural network (NN) classifier on the U.S. state of Iowa to classify crops into corn/soybean/other using moderate resolution imaging spectroradiometer (MODIS) data. USDA cropland data layers were aggregated to the 250-m resolution of MODIS and used as ground truth, based on a cropland mask from the National Land Cover Database. Results show that the spatial-prior-adjustment method, which predicts prior probabilities for low-confidence pixels based on class percentages of initial NN classification, increased overall accuracy of low-confidence pixels between 2% and 3.3% over the standard NN classification. The temporal-prior-adjustment method, which uses crop classes from the previous six years to estimate prior probabilities for the current year, shows significantly greater accuracy improvement for low-confidence pixels (almost 7%) over the standard NN classification. Increased benefit of the temporal-prior-adjustment method relative to the spatial-prior-adjustment method is likely due to increased information from more ground truth data (from previous years) than the spatial method.

A systematic study of the class imbalance problem in convolutional neural networks

In this study, we systematically investigate the impact of class imbalance on classification performance of convolutional neural networks (CNNs) and compare frequently used methods to address the issue. Class imbalance is a common problem that has been comprehensively studied in classical machine learning, yet very limited systematic research is available in the context of deep learning. In our study, we use three benchmark datasets of increasing complexity, MNIST, CIFAR-10 and ImageNet, to investigate the effects of imbalance on classification and perform an extensive comparison of several methods to address the issue: oversampling, undersampling, two-phase training, and thresholding that compensates for prior class probabilities. Our main evaluation metric is area under the receiver operating characteristic curve (ROC AUC) adjusted to multi-class tasks since overall accuracy metric is associated with notable difficulties in the context of imbalanced data. Based on results from our experiments we conclude that (i) the effect of class imbalance on classification performance is detrimental; (ii) the method of addressing class imbalance that emerged as dominant in almost all analyzed scenarios was oversampling; (iii) oversampling should be applied to the level that completely eliminates the imbalance, whereas the optimal undersampling ratio depends on the extent of imbalance; (iv) as opposed to some classical machine learning models, oversampling does not cause overfitting of CNNs; (v) thresholding should be applied to compensate for prior class probabilities when overall number of properly classified cases is of interest.

Optimizing community-level surveillance data for pediatric asthma management.

Community-level approaches for pediatric asthma management rely on locally collected information derived primarily from two sources: claims records and school-based surveys. We combined claims and school-based surveillance data, and examined the asthma-related risk patterns among adolescent students.Symptom data collected from school-based asthma surveys conducted in Oakland, CA were used for case identification and determination of severity levels for students (high and low). Survey data were matched to Medicaid claims data for all asthma-related health care encounters for the year prior to the survey. We then employed recursive partitioning to develop classification trees that identified patterns of demographics and healthcare utilization associated with severity.A total of 561 students had complete matched data; 86.1% were classified as high-severity, and 13.9% as low-severity asthma. The classification tree consisted of eight subsets: three indicating high severity and five indicating low severity. The risk subsets highlighted varying combinations of non-specific demographic and socioeconomic predictors of asthma prevalence, morbidity and severity. For example, the subset with the highest class-prior probability (92.1%) predicted high-severity asthma and consisted of students without prescribed rescue medication, but with at least one in-clinic nebulizer treatment. The predictive accuracy of the tree-based model was approximately 66.7%, with an estimated 91.1% of high-severity cases and 42.3% of low-severity cases correctly predicted.Our analysis draws on the strengths of two complementary datasets to provide community-level information on children with asthma, and demonstrates the utility of recursive partitioning methods to explore a combination of features that convey asthma severity.

Investigation on Land Cover Mapping Capability of Maximum Likelihood Classifier: A Case Study on North Canara, India

The success of a large number of real-world applications depends on the efficiency with which land use land cover classes are extracted from Remotely Sensed (RS) imagery. This study aims at analysing the performance of Maximum Likelihood Classifier (MLC) over multispectral remote sensor data characterized by the overlapping of spectral classes and presence of mixed pixels, investigating the impact of prior class probabilities on the overall classification performance, the impact of the number of land cover classes on classification accuracy, and identifying the decision parameters for assigning a pixel to a land cover class. A quantitative analysis of decision parameters of MLC for pixel assignment is presented for five randomly selected pixels in the study area. Landsat 8 multispectral data of North Canara District was collected from USGS website and is considered for the research. Seven land use land cover classes were identified over the study area. Results obtained indicate that use of prior probability estimates improves the classification accuracy. The study also presents the variation in classification accuracy for four different cases of total number of land cover classes. The conducted research is useful in understanding the performance of MLC under multiple scenarios.

An empirical study on the effect of imbalanced data on bleeding detection in endoscopic video.

In biomedical applications including classification of endoscopic videos, class imbalance is a common problem arising from the significant difference between the prior probabilities of different classes. In this paper, we investigate the performance of different classifiers for varying training data distribution in case of bleeding detection problem through three experiments. In the first experiment, we analyze the classifier performance for different class distribution with a fixed sized training dataset. The experiment provides the indication of the required class distribution for optimum classification performance. In the second and third experiments, we investigate the effect of both training data size and class distribution on the classification performance. From our experiments, we found that a larger dataset with moderate class imbalance yields better classification performance compared to a small dataset with balanced distribution. Ensemble classifiers are more robust to the variation in training dataset compared to single classifier.

Extended least squares support vector machines for ordinal regression

We extend LS-SVM to ordinal regression, which has wide applications in many domains such as social science and information retrieval where human-generated data play an important role. Most current methods based on SVM for ordinal regression suffer from the problem of ignoring the distribution information reflected by the samples clustered around the centers of each class. This problem would degrade the performance of SVM-based methods since the classifiers only depend on the scattered samples on the border which induce large margin. Our method takes the samples clustered around class centers into account and has a competitive computational complexity. Moreover, our method would easily produce the optimal cut-points according to the prior class probabilities and hence may obtain more reasonable results when the prior class probabilities are not the same. Experiments on simulated datasets and benchmark datasets, especially on the real ordinal datasets, demonstrate the effectiveness of our method.

Subspace Learning via Local Probability Distribution for Hyperspectral Image Classification

The computational procedure of hyperspectral image (HSI) is extremely complex, not only due to the high dimensional information, but also due to the highly correlated data structure. The need of effective processing and analyzing of HSI has met many difficulties. It has been evidenced that dimensionality reduction has been found to be a powerful tool for high dimensional data analysis. Local Fisher’s liner discriminant analysis (LFDA) is an effective method to treat HSI processing. In this paper, a novel approach, called PD-LFDA, is proposed to overcome the weakness of LFDA. PD-LFDA emphasizes the probability distribution (PD) in LFDA, where the maximum distance is replaced with local variance for the construction of weight matrix and the class prior probability is applied to compute the affinity matrix. The proposed approach increases the discriminant ability of the transformed features in low dimensional space. Experimental results on Indian Pines 1992 data indicate that the proposed approach significantly outperforms the traditional alternatives.