Class-conditional Probability Research Articles

Adaptive evidential K-NN classification: Integrating neighborhood search and feature weighting

The number of nearest neighbors K and the utilized distance measure considerably impact the performance of the K-nearest neighbor (K-NN) algorithm. The information provided by neighbors may be perplexing due to the set presumption of K for every test sample without any prior knowledge, which might result in incorrect classification results. The hypothesis of locally constant class conditional probabilities relied on by K-NN, is violated by high dimensions, which also bring out the dimensionality curse. Notably, in the case of imperfect data, training samples may be noise-corrupted or located in substantially overlapping regions, thus impairing the effectiveness of the K-NN rule. In contrast to existing research, which frequently focuses on just one of the problems above, this paper presents an adaptive evidential K-NN classification (AEK-NN) algorithm that synchronizes neighborhood search with feature weighting. In order to jointly learn the adaptive neighborhood for each sample and various feature weights, AEK-NN maps the Euclidean space into a rebuilt space consisting of similarity coefficients through a two-stage training process. Within the evidence theory framework, AEK-NN produces classification results for data with imperfect labels. The advantages of applying evidence theory, incorporating adaptive neighborhood search, and feature weighting are shown through ablation studies. We also use simulated and real-world datasets to demonstrate the state-of-the-art performance of AEK-NN.

Distribution shift alignment in visual domain adaptation

Domain adaptation provides the possibility of utilizing the knowledge gained from an auxiliary domain to accomplish the task in another related domain. In this paper, a common feature representation of these domains is learned through a new proposed idea to reduce inter-domain differences more precisely which results in higher accuracy for unsupervised domain adaptation. To decrease these divergences, the proposed method finds a subspace that reduces data distribution discrepancy between domains. To achieve this goal, minimization of the divergence of the class conditional probability distributions is considered. By considering class conditional distributions, more discriminative information of data is preserved compared to applying marginal distributions. Also, an explicit parametric distribution of the source and target domains is considered to reduce the discrepancy between the two domains data which results in higher accuracy compared to the other relevant domain adaptation methods. Experimental studies on benchmark image classification tasks confirmed our assumptions and indicated the significant improvement given by the proposed method compared to the other state-of-the-art methods.

Optimization of Probability Density Functions Applicable for Hourly Rainfall

In order to improve the calculation accuracy of the rainfall probability distribution in related applications, this study aimed to select a theoretical function from applicable functions for three classes of the class-conditional probability density function (CCPD) of hourly rainfall series. The three applicable functions are generalized gamma distribution (GΓD), generalized normal distribution (GND), and Weibull distribution. For the reason that it is hard to distinguish the advantages and disadvantages of the three functions by the probability plot and error analysis of fitted values, optimization criteria are proposed, which are the Bayesian information criterion (BIC) and the estimated accuracy of both the annual average rainfall (AAR) and the annual average continuous rainfall (AACR). The results show that by using three applicable functions in 15 regions, the relative fitting deviations for CCPD1 were less than 2.3% and less than 3.3% for ln(CCPD1). The goodness-of-fit values were all above 0.98 for CCPD1 and greater than 0.94 for ln(CCPD1). The fitting effect of the Weibull distribution was relatively poor from the perspective of the probability plot and error analysis of the fitted values, while the three applicable functions could be used to fit CCPD. GΓD had the highest fitting accuracy for the three classes of CCPDs, but there is concern about overfitting due to its broad spectrum. GND, with fewer parameters, had comparable performance to GΓD, and when fitting CCPD1 using GND, the mean relative fitting deviation was 0.6%, the coefficient of determination was 0.999, and for ln(CCPD1), they were 1.45% and 0.989. At the same time, GND performed well in estimating the AARs, with an 8.6% relative error and a 0.92 correlation coefficient in the fifteen regions, indicating that GND can well reflect the spatial variation characteristics of the AAR. Moreover, the function form of GND is simple. GND follows the parsimonious principle, and it is suitable for the whole domain. Therefore, GND is recommended as the theoretical density function based on the optimization criteria. The genetic algorithm was adopted to obtain the approximate solution of the parameters through optimization, which can simplify the derivation and calculation steps. The multiplicative and additive fitting errors were both used in the objective functions, which gave comprehensive consideration to both ends of the fitting curve.

Function applicability to class conditional probability density of hourly rainfall

暴雨强度公式在水文、气象、工程设计等各领域都是非常关注的问题,而常用降雨概率分布函数的适用性欠缺,理论分布函数一直处于争鸣之中。从逐时降雨概率密度函数的适用性分析入手,有利于发现普适且恰当的理论密度函数。本文从我国暴雨洪涝灾害易发区中沿30°N选取4个经纬度长方形区域(雅安附近、鄂西南、江汉平原南部、杭州湾西),并在其南、北各选一对比分区(海南岛、郑州),对6个分区内降雨资料直接采用全样本,统计逐时降雨的三类概率密度经验函数,对照这些函数的特性,从理论上分析了众多分布函数的适用性,筛选适用函数并进行拟合试验,优选出理论密度函数。研究结果表明:三参广义伽玛函数拟合误差最小,而两参广义正态函数更恰当、被首推为理论密度函数;拟合参数寻优时的目标函数综合了乘性与加性误差模型,能使拟合曲线兼顾头尾;本研究有别于极值降雨概率分布中仅采用极少部分样本的方法,采用降雨概率密度方法替代传统的年极值法,使重现期计算更准确有效,能提高暴雨强度公式的科学性,拟合的高精度与函数的普适性有望解决降雨概率分布模型统一的问题。;Due to the deficiency of international-unified theoretical function for rainfall probability distribution, it is necessary to analyze the applicability, appropriateness and universality of commonly used probability density function(PDF). On the basis of summarizing the characteristics of three kinds of class conditional probability density of hourly rainfall, the applicability of 20 kinds of functions is theoretically analyzed and compared. The generalized Gamma distribution(GΓD), generalized normal distribution(GND) and Weibull distributions are selected as reference functions, and the genetic algorithm and genetic algorithms are used to carry out fitting experiments. When the fitting deviation is equivalent to the observation error, a necessary condition for the appropriateness of the theoretical distribution function is used for discriminant analysis. The fitting results show that the Weibull distribution performs worst, the GΓD has the smallest fitting deviation, and the GND performs best. Therefore the GND is recommended for theoretical distribution fitting. In addition, the genetic algorithm can directly obtain the optimal approximate solution through multi-objective parameter optimization, which can simplify the derivation and calculation steps. The multiplicative and additive deviation models are both used as objective functions, which can take the beginning and the end of fitting curves, as well as the absolute and relative deviations into account.

A proxy learning curve for the Bayes classifier

In this paper, a theoretical learning curve is derived for the multi-class Bayes classifier. This curve fits general multivariate parametric models of the class-conditional probability density. The derivation uses a proxy approach based on analyzing the convergence of a statistic which is proportional to the posterior probability of the true class. By doing so, the curve depends only on the training set size and on the dimension of the feature vector; it does not depend on the model parameters. Essentially, the learning curve provides an estimate of the reduction in the excess of the probability of error that can be obtained by increasing the training set size. This makes it attractive in order to deal with the practical problems of defining appropriate training set sizes.

ImAiR: Airwriting Recognition Framework Using Image Representation of IMU Signals

The problem of Airwriting Recognition is focused on identifying letters written by movement of finger in free space. It is a type of gesture recognition where the dictionary corresponds to letters in a specific language. In particular, airwriting recognition using sensor data from wrist-worn devices can be used as a medium of user input for applications in Human-Computer Interaction (HCI). Recognition of in-air trajectories using such wrist-worn devices is limited in literature and forms the basis of the current work. In this paper, we propose an airwriting recognition framework by first encoding the time-series data obtained from a wearable Inertial Measurement Unit (IMU) on the wrist as images and then utilizing deep learning-based models for identifying the written alphabets. The signals recorded from 3-axis accelerometer and gyroscope in IMU are encoded as images using different techniques such as Self Similarity Matrix (SSM), Gramian Angular Field (GAF) and Markov Transition Field (MTF) to form two sets of 3-channel images. These are then fed to two separate classification models and letter prediction is made based on an average of the class conditional probabilities obtained from the two models. Several standard model architectures for image classification such as variants of ResNet, DenseNet, VGGNet, AlexNet and GoogleNet have been utilized. Experiments performed on two publicly available datasets demonstrate the efficacy of the proposed strategy. The code for our implementation will be made available at https://github.com/ayushayt/ImAiR.

Sparse concordance‐based ordinal classification

AbstractOrdinal classification is an important area in statistical machine learning, where labels exhibit a natural order. One of the major goals in ordinal classification is to correctly predict the relative order of instances. We develop a novel concordance‐based approach to ordinal classification, where a concordance function is introduced and a penalized smoothed method for optimization is designed. Variable selection using the penalty is incorporated for sparsity considerations. Within the set of classification rules that maximize the concordance function, we find optimal thresholds to predict labels by minimizing a loss function. After building the classifier, we derive nonparametric estimation of class conditional probabilities. The asymptotic properties of the estimators as well as the variable selection consistency are established. Extensive simulations and real data applications show the robustness and advantage of the proposed method in terms of classification accuracy, compared with other existing methods.

A novel dependency-oriented mixed-attribute data classification method

How to design an efficient method to handle mixed-attribute data classification (MADC) problems has become a hot topic in data mining and machine learning. Current MADC methods mostly transform mixed-attribute data into discrete-attribute data or continuous-attribute data before classification algorithms are trained. The discretization of continuous-attribute data usually results in information loss, while the binarization of discrete-attribute data generally yield more discrete-attributes. To address these issues, this paper proposes a novel MADC method abbreviated as DO-RVFL-NBC, which is a Dependency-Oriented aggregation model of random vector functional link (RVFL) network and naive Bayes classifier (NBC). First, the method transforms the original mixed-attribute set into a dependent attribute set and an independent attribute set by considering the variation rates of dependence and independence, respectively. Second, a RVFL network is trained based on the dependent attribute set where each attribute has a weight to represent its dependence importance degree. Third, a weighted NBC is constructed by assigning the independence importance degrees as weights for the calculation of class-conditional probability. Finally, exhaustive experiments are conducted to validate the feasibility, rationality, and effectiveness of the DO-RVFL-NBC method using 22 benchmark mixed-attribute data sets. Experimental results show that (1) dependence and independence exist in the original mixed-attribute set and can be effectively explored; (2) changes of attribute dependences can improve the generalization capabilities of the RVFL network and NBC; and (3) a statistical analysis indicates that DO-RVFL-NBC can obtain considerably better testing accuracies on the benchmark mixed-attribute data sets in comparison with 13 other MADC methods. This demonstrates that DO-RVFL-NBC is a viable approach for MADC problems.

Radar Cross Section Based Statistical Recognition of UAVs at Microwave Frequencies

This article presents a radar cross-section (RCS)-based statistical recognition system for identifying/classifying unmanned aerial vehicles (UAVs) at microwave frequencies. First, the article presents the results of the vertical (VV) and horizontal (HH) polarization RCS measurement of six commercial UAVs at 15 and 25 GHz in a compact range anechoic chamber. The measurement results show that the average RCS of the UAVs depends on shape, size, material composition of the target UAV as well as the azimuth angle, frequency, and polarization of the illuminating radar. Afterward, radar characterization of the target UAVs is achieved by fitting the RCS measurement data to 11 different statistical models. From the model selection analysis, we observe that the lognormal, generalized extreme value, and gamma distributions are most suitable for modeling the RCS of the commercial UAVs while the Gaussian distribution performed relatively poorly. The best UAV radar statistics forms the class conditional probability densities for the proposed UAV statistical recognition system. The performance of the UAV statistical recognition system is evaluated at different signal noise ratio (SNR) with the aid of Monte Carlo analysis. At an SNR of 10 dB, the average classification accuracy of 97.60% or better is achievable.

An Improved K-Nearest Neighbor Algorithm for Pattern Classification

This paper proposed a “Locally Adaptive K-Nearest Neighbor (LAKNN) algorithm” for pattern exploration problem to enhance the obscenity of dimensionality. To compute neighborhood local linear discriminant analysis is an effective metric which determines the local decision boundaries from centroid information. KNN is a novel approach which uses in many classifications problem of data mining and machine learning. KNN uses class conditional probabilities for unfamiliar pattern. For limited training data in high dimensional feature space this hypothesis is unacceptable due to disfigurement of high dimensionality. To normalize the feature value of dissimilar metrics, Standard Euclidean Distance is used in KNN which s misguide to find a proper subset of nearest points of the pattern to be predicted. To overcome the effect of high dimensionality LANN uses a new variant of Standard Euclidian Distance Metric. A flexible metric is estimated for computing neighborhoods based on Chi-squared distance analysis. Chi-squared metric is used to ascertains most significant features in finding k-closet points of the training patterns. This paper also shows that LANN outperformed other four different models of KNN and other machine-learning algorithm in both training and accuracy.

Convergence rates of deep ReLU networks for multiclass classification

For classification problems, trained deep neural networks return probabilities of class memberships. In this work we study convergence of the learned probabilities to the true conditional class probabilities. More specifically we consider sparse deep ReLU network reconstructions minimizing cross-entropy loss in the multiclass classification setup. Interesting phenomena occur when the class membership probabilities are close to zero. Convergence rates are derived that depend on the near-zero behaviour via a margin-type condition.

Correcting the Estimation of Viral Taxa Distributions in Next-Generation Sequencing Data after Applying Artificial Neural Networks.

Estimating the taxonomic composition of viral sequences in a biological samples processed by next-generation sequencing is an important step in comparative metagenomics. Mapping sequencing reads against a database of known viral reference genomes, however, fails to classify reads from novel viruses whose reference sequences are not yet available in public databases. Instead of a mapping approach, and in order to classify sequencing reads at least to a taxonomic level, the performance of artificial neural networks and other machine learning models was studied. Taxonomic and genomic data from the NCBI database were used to sample labelled sequencing reads as training data. The fitted neural network was applied to classify unlabelled reads of simulated and real-world test sets. Additional auxiliary test sets of labelled reads were used to estimate the conditional class probabilities, and to correct the prior estimation of the taxonomic distribution in the actual test set. Among the taxonomic levels, the biological order of viruses provided the most comprehensive data base to generate training data. The prediction accuracy of the artificial neural network to classify test reads to their viral order was considerably higher than that of a random classification. Posterior estimation of taxa frequencies could correct the primary classification results.

Efficient set-valued prediction in multi-class classification

In cases of uncertainty, a multi-class classifier preferably returns a set of candidate classes instead of predicting a single class label with little guarantee. More precisely, the classifier should strive for an optimal balance between the correctness (the true class is among the candidates) and the precision (the candidates are not too many) of its prediction. We formalize this problem within a general decision-theoretic framework that unifies most of the existing work in this area. In this framework, uncertainty is quantified in terms of conditional class probabilities, and the quality of a predicted set is measured in terms of a utility function. We then address the problem of finding the Bayes-optimal prediction, i.e., the subset of class labels with the highest expected utility. For this problem, which is computationally challenging as there are exponentially (in the number of classes) many predictions to choose from, we propose efficient algorithms that can be applied to a broad family of utility functions. Our theoretical results are complemented by experimental studies, in which we analyze the proposed algorithms in terms of predictive accuracy and runtime efficiency.

Fast convergence rates of deep neural networks for classification

We derive the fast convergence rates of a deep neural network (DNN) classifier with the rectified linear unit (ReLU) activation function learned using the hinge loss. We consider three cases for a true model: (1) a smooth decision boundary, (2) smooth conditional class probability, and (3) the margin condition (i.e., the probability of inputs near the decision boundary is small). We show that the DNN classifier learned using the hinge loss achieves fast rate convergences for all three cases provided that the architecture (i.e., the number of layers, number of nodes and sparsity) is carefully selected. An important implication is that DNN architectures are very flexible for use in various cases without much modification. In addition, we consider a DNN classifier learned by minimizing the cross-entropy, and show that the DNN classifier achieves a fast convergence rate under the conditions that the noise exponent and margin exponent are large. Even though they are strong, we explain that these two conditions are not too absurd for image classification problems. To confirm our theoretical explanation, we present the results of a small numerical study conducted to compare the hinge loss and cross-entropy.

Guided Zoom: Zooming into Network Evidence to Refine Fine-Grained Model Decisions.

In state-of-the-art deep single-label classification models, the top- k (k=2,3,4, ...) accuracy is usually significantly higher than the top-1 accuracy. This is more evident in fine-grained datasets, where differences between classes are quite subtle. Exploiting the information provided in the top k predicted classes boosts the final prediction of a model. We propose Guided Zoom, a novel way in which explainability could be used to improve model performance. We do so by making sure the model has "the right reasons" for a prediction. The reason/evidence upon which a deep neural network makes a prediction is defined to be the grounding, in the pixel space, for a specific class conditional probability in the model output. Guided Zoom examines how reasonable the evidence used to make each of the top- k predictions is. Test time evidence is deemed reasonable if it is coherent with evidence used to make similar correct decisions at training time. This leads to better informed predictions. We explore a variety of grounding techniques and study their complementarity for computing evidence. We show that Guided Zoom results in an improvement of a model's classification accuracy and achieves state-of-the-art classification performance on four fine-grained classification datasets. Our code is available at https://github.com/andreazuna89/Guided-Zoom.

New multivariate kernel density estimator for uncertain data classification

Uncertainty in data occurs in diverse applications due to measurement errors, data incompleteness, and multiple repeated measurements. Several classifiers for uncertain data have been developed to tackle this uncertainty. However, the existing classifiers do not consider the dependencies among uncertain features, even though this dependency has a critical effect on classification accuracy. Therefore, we propose a new Bayesian classification model that considers the correlation among uncertain features. To handle the uncertainty of data, new multivariate kernel density estimators are developed to estimate the class conditional probability density function of categorical, continuous, and mixed uncertain data. Experimental results with simulated data and real-life data sets show that the proposed approach is better than the existing approaches for classification of uncertain data in terms of classification accuracy.

Simultaneously Spatiospectral Pattern Learning and Contaminated Trial Pruning for Electroencephalography-Based Brain Computer Interface

Electroencephalography (EEG)-based brain computer interfaces (BCIs) translate motor imagery commands into the movements of an external device (e.g., a robotic arm). The automatic design of spectral and spatial filters is a challenging task, as the frequency bands of the spectral filters must be predefined by previously published studies and given that they may be affected during trials by artifacts and improper motor imagery (MI). This study aimed to eliminate the contaminated trials automatically during classifier training, and to simultaneously learn the spectral and spatial patterns without the need for predefined frequency bands. Compared with previous studies that measured the discriminative power of a frequency band based on mutual information, this study determined the difference of the class conditional probability density function between two MI classes. This information was further shared to measure the contamination level of the trial that simplified the computation. A particle-based approximation technique iteratively constructed a filter bank that extracted discriminative features, and simultaneously removed potentially contaminated trials. The particle weight was estimated by an analysis of variance F-test instead of mutual information as commonly used in previous studies. The experimental results of a publicly available dataset revealed that the proposed method outperformed the other BCI in terms of the classification accuracy. Asymmetrical spatial patterns were found on left- versus right-hand MI classifications. The learnt spectral and spatial patterns were consistent with prior neurophysiological knowledge.

A dynamic ensemble outlier detection model based on an adaptive k-nearest neighbor rule

Ensembles of outlier detectors are drawing increasing attentions recently, in spite of the difficulty on developing ensembles in the framework of unsupervised learning. We have noted that existing outlier ensembles often use certain fusion rules (e.g. majority voting) to aggregate individual learners. Theoretically, these individuals are assumed to be error-independent so that single models can be outperformed by the ensemble. However, it is of great difficulty to satisfy this assumption in practical applications. By dynamic selecting more competent individual(s) for each test pattern, this problem can be alleviated effectively. Inspired by this idea, this paper proposes a dynamic ensemble outlier detection model using one-class classifiers as base learners. As the competences of base detectors are estimated totally on data points in the validation set, its impact on the selection is significant. In order to achieve an efficient selection, we propose an adaptive k-nearest neighbor (KNN) rule, instead of traditional KNN algorithm, to constitute the validation set for each test pattern. Our adaptive KNN rule firstly uses algorithm support vector data description (SVDD) to mine the local area where class conditional probabilities are not constant in terms of the corresponding test pattern. Competences estimated with neighbor patterns in this area should thus be more accurate than that by traditional KNN rule. A probabilistic model that uses posterior probabilities of one-class classifiers is used then to estimate classifier competences. We present experimental evidence of the detection performance improvement over single models and over a variety of static ensemble models, by using data sets from UCI repository.

SPAM-Net: A CNN-Based SAR Target Recognition Network With Pose Angle Marginalization Learning

Recently, deep convolutional neural networks (CNNs) have started to be applied for automatic target recognition (ATR) problems of synthetic aperture radar (SAR) images. In conventional SAR-ATR algorithms, the pose angle information of the target has been importantly used. However, recent deep learning-based SAR-ATR algorithms often only utilize the intensity information. In this paper, based on the prior works that the pose angle is an important latent variable for boosting target recognition performance, we propose a CNN-based SAR target recognition network with pose angle marginalization learning, called SPAM-Net that marginalizes the conditional probabilities of SAR targets over their pose angles to precisely estimate the true class probabilities. The proposed SPAM-Net consists of two sub-nets: (i) a sub-net for class-conditional probability estimation, called CP sub-net, and (ii) a sub-net for pose angle probability estimation, called PP sub-net. The two sub-nets are jointly learned via an end-to-end manner in a Bayesian framework so that the SPAM-Net incorporates the pose angle information into target recognition task effectively. The SPAM-Net outperforms our baseline network that does not utilize the pose angle information. In the experiments, we intensively analyze the effectiveness of pose angle information for SAR-ATR, revealing that more accurate pose angle information helps the SPAM-Net precisely estimate target classes for the misclassified target groups that are obtained by the baseline network. Furthermore, our method also outperforms the other state-of-the-art SAR-ATR algorithms, yielding the correct target recognition rate with average 99.61%.

Learning from Positive and Unlabeled Data without Explicit Estimation of Class Prior

Learning a classifier from positive and unlabeled data may occur in various applications. It differs from the standard classification problems by the absence of labeled negative examples in the training set. So far, two main strategies have typically been used for this issue: the likely negative examplesbased strategy and the class prior-based strategy, in which the likely negative examples or the class prior is required to be obtained in a preprocessing step. In this paper, a new strategy based on the Bhattacharyya coefficient is put forward, which formalizes this learning problem as an optimization problem and does not need a preprocessing step. We first show that with the given positive class conditional probability density function (PDF) and the mixture PDF of both the positive class and the negative class, the class prior can be estimated by minimizing the Bhattacharyya coefficient of the positive class with respect to the negative class. We then show how to use this result in an implicit mixture model of restricted Boltzmann machines to estimate the positive class conditional PDF and the negative class conditional PDF directly to obtain a classifier without the explicit estimation of the class prior. Many experiments on real and synthetic datasets illustrated the superiority of the proposed approach.