AUC Maximization Research Articles

Weakly Supervised AUC Optimization: A Unified Partial AUC Approach.

Since acquiring perfect supervision is usually difficult, real-world machine learning tasks often confront inaccurate, incomplete, or inexact supervision, collectively referred to as weak supervision. In this work, we present WSAUC, a unified framework for weakly supervised AUC optimization problems, which covers noisy label learning, positive-unlabeled learning, multi-instance learning, and semi-supervised learning scenarios. Within the WSAUC framework, we first frame the AUC optimization problems in various weakly supervised scenarios as a common formulation of minimizing the AUC risk on contaminated sets, and demonstrate that the empirical risk minimization problems are consistent with the true AUC. Then, we introduce a new type of partial AUC, specifically, the reversed partial AUC (rpAUC), which serves as a robust training objective for AUC maximization in the presence of contaminated labels. WSAUC offers a universal solution for AUC optimization in various weakly supervised scenarios by maximizing the empirical rpAUC. Theoretical and experimental results under multiple settings support the effectiveness of WSAUC on a range of weakly supervised AUC optimization tasks.

Multi-class AUC maximization for imbalanced ordinal multi-stage tropical cyclone intensity change forecast

Intense tropical cyclones (TCs) cause significant damage to human societies. Forecasting the multiple stages of TC intensity changes is considerably crucial yet challenging. This difficulty arises due to imbalanced data distribution and the need for ordinal multi-class classification. While existing classification methods, such as linear discriminant analysis, have been utilized to predict rare rapidly intensifying (RI) stages based on features related TC intensity changes, they are limited to binary classification distinguishing between RI and non-RI stages. In this paper, we introduce a novel methodology to tackle the challenges of imbalanced ordinal multi-class classification. We extend the Area Under the Curve maximization technique with inter-instance/class cross-hinge losses and inter-class distance-based slack variables. The proposed loss function, implemented within a deep learning framework, demonstrates its effectiveness using real sequence data of multi-stage TC intensity changes, including satellite infrared images and environmental variables observed in the western North Pacific.

GTE: a graph learning framework for prediction of T-cell receptors and epitopes binding specificity.

The interaction between T-cell receptors (TCRs) and peptides (epitopes) presented by major histocompatibility complex molecules (MHC) is fundamental to the immune response. Accurate prediction of TCR-epitope interactions is crucial for advancing the understanding of various diseases and their prevention and treatment. Existing methods primarily rely on sequence-based approaches, overlooking the inherent topology structure of TCR-epitope interaction networks. In this study, we present $GTE$, a novel heterogeneous Graph neural network model based on inductive learning to capture the topological structure between TCRs and Epitopes. Furthermore, we address the challenge of constructing negative samples within the graph by proposing a dynamic edge update strategy, enhancing model learning with the nonbinding TCR-epitope pairs. Additionally, to overcome data imbalance, we adapt the Deep AUC Maximization strategy to the graph domain. Extensive experiments are conducted on four public datasets to demonstrate the superiority of exploring underlying topological structures in predicting TCR-epitope interactions, illustrating the benefits of delving into complex molecular networks. The implementation code and data are available at https://github.com/uta-smile/GTE.

Limited Memory Online Gradient Descent for Kernelized Pairwise Learning with Dynamic Averaging

Pairwise learning, an important domain within machine learning, addresses loss functions defined on pairs of training examples, including those in metric learning and AUC maximization. Acknowledging the quadratic growth in computation complexity accompanying pairwise loss as the sample size grows, researchers have turned to online gradient descent (OGD) methods for enhanced scalability. Recently, an OGD algorithm emerged, employing gradient computation involving prior and most recent examples, a step that effectively reduces algorithmic complexity to O(T), with T being the number of received examples. This approach, however, confines itself to linear models while assuming the independence of example arrivals. We introduce a lightweight OGD algorithm that does not require the independence of examples and generalizes to kernel pairwise learning. Our algorithm builds the gradient based on a random example and a moving average representing the past data, which results in a sub-linear regret bound with a complexity of O(T). Furthermore, through the integration of O(√T logT) random Fourier features, the complexity of kernel calculations is effectively minimized. Several experiments with real-world datasets show that the proposed technique outperforms kernel and linear algorithms in offline and online scenarios.

FAUC-S: Deep AUC maximization by focusing on hard samples

Deep AUC maximization (DAM) is a popular method to deal with complex imbalanced classification problems. It learns a deep neural network by minimizing a surrogate AUC loss. The mostly used surrogates are the pairwise square loss and its variants. However, they are usually used in regression problems and are sensitive to noise samples. The easy samples suffer larger losses than those without correct classification, which is unreasonable in optimization problems. In this paper, we propose a focus AUC loss based on samples (FAUC-S) by constructing a differentiable weight function that identifies hard and easy samples, which makes easy samples have small losses, and hard samples have large losses. It is more reasonable than the traditional AUC loss while having the same advantages on large-scale datasets. In experiments, we use an end-to-end compositional DAM to investigate the effect of different weight functions on the AUC loss and compare it with other methods. Experimental results on several benchmark datasets demonstrate that FAUC-S achieves superior performance in terms of AUC compared to existing state-of-the-art methods.

Learning Rates for Nonconvex Pairwise Learning

Pairwise learning is receiving increasing attention since it covers many important machine learning tasks, e.g., metric learning, AUC maximization, and ranking. Investigating the generalization behavior of pairwise learning is thus of great significance. However, existing generalization analysis mainly focuses on the convex objective functions, leaving the nonconvex pairwise learning far less explored. Moreover, the current learning rates of pairwise learning are mostly of slower order. Motivated by these problems, we study the generalization performance of nonconvex pairwise learning and provide improved learning rates. Specifically, we develop different uniform convergence of gradients for pairwise learning under different assumptions, based on which we characterize empirical risk minimizer, gradient descent, and stochastic gradient descent. We first establish learning rates for these algorithms in a general nonconvex setting, where the analysis sheds insights on the trade-off between optimization and generalization and the role of early-stopping. We then derive faster learning rates of order O(1/n) for nonconvex pairwise learning with a gradient dominance curvature condition, where n is the sample size. Provided that the optimal population risk is small, we further improve the learning rates to O(1/n2), which, to the best of our knowledge, are the first O(1/n2) rates for pairwise learning.

A Novel AUC Maximization Imbalanced Learning Approach for Predicting Composite Outcomes in COVID-19 Hospitalized Patients.

The COVID-19 patient data for composite outcome pre- diction often come with class imbalance issues, i.e., only a small group of patients develop severe composite events after hospital admission while the rest do not. An ideal COVID-19 composite outcomes prediction model should seek strong imbalanced learning capability. Such a model also should have fewer tuning hyperparameters to preserve its good usability and exhibit its potential for fast incremen- tal learning. Towards this goal, by means of the classical proximal support vector machine (PSVM), this study pro- poses a novel imbalanced learning approach called Imbalanced maximizing-Area Under the Curve (AUC) Proximal Support Vector Machine (ImAUC-PSVM) to predict the hospitalized COVID-19 patients' composite outcomes within 30 days of hospitalization. ImAUC-PSVM has the following merits: (1) it incorporates the straightforward AUC maximization into the objective function and hence has fewer parameters to tune. This makes it naturally suitable for handling imbalanced COVID-19 data with a simplified training process. (2) Our theoretical derivations reveal that it has the same form of the analytical solution as PSVM. Therefore, it inherits the advantages of PSVM in dealing with incremental COVID- 19 cases using the fast incremental updating. Our proposed classifier was built and validated internally and externally on the real COVID-19 patient data retrieved from three separate sites of Mayo Clinic in the United States and also validated on the public datasets using various performance metrics. Experi- mental results demonstrate that ImAUC-PSVM outperforms in most cases, showing its potential to assist clinicians in triaging COVID-19 patients at early stage in the hospital settings.

Efficient fraud detection using deep boosting decision trees

Fraud detection is to identify, monitor, and prevent potentially fraudulent activities from complex data. The recent development and success in AI, especially machine learning, provides a new data-driven way to deal with fraud. From a methodological point of view, machine learning based fraud detection can be divided into two categories, i.e., conventional methods (decision tree, tree boosting methods, etc.) and deep learning, both of which have significant limitations in terms of the lack of representation learning ability for the former and interpretability for the latter. Furthermore, due to the rarity of detected fraud cases, the associated data is usually imbalanced, which seriously degrades the performance of classification algorithms.In this paper, we propose deep boosting decision trees (DBDT), a novel approach for fraud detection based on gradient boosting and neural networks. In order to combine the advantages of both conventional methods and deep learning, we first construct soft decision tree (SDT), a decision tree structured model with neural networks as its nodes, and then ensemble SDTs using the idea of gradient boosting. In this way we embed neural networks into gradient boosting to improve its representation learning capability and meanwhile maintain the interpretability. Furthermore, aiming at the rarity of detected fraud cases, in the model training phase we propose a compositional AUC maximization approach to deal with data imbalances at algorithm level. Extensive experiments on several real-life fraud detection datasets show that DBDT can significantly improve the performance and meanwhile maintain good interpretability. Our code is available at https://github.com/freshmanXB/DBDT.

Distributed spectral pairwise ranking algorithms

This paper considers spectral pairwise ranking algorithms in a reproducing kernel Hilbert space. The concerned algorithms include a large family of regularized pairwise learning algorithms. Motivated by regularization methods, spectral algorithms are proposed to solve ill-posed linear inverse problems, then developed in learning theory and inverse problems. Recently, pairwise learning tasks such as bipartite ranking, similarity metric learning, Minimum Error Entropy principle, and AUC maximization have received increasing attention due to their wide applications. However, the spectral algorithm acts on the spectrum of the empirical integral operator or kernel matrix, involving the singular value decomposition or the inverse of the matrix, which is time-consuming when the sample size is immense. Our contribution is twofold. First, under some general source conditions and capacity assumptions, we establish the first-ever mini-max optimal convergence rates for spectral pairwise ranking algorithms. Second, we consider the distributed version of the algorithms based on a divide-and-conquer approach and show that, as long as the partition of the data set is not too large, the distributed learning algorithm enjoys both computational efficiency and statistical optimality.

AUC Maximization in the Era of Big Data and AI: A Survey

Area under the ROC curve, a.k.a. AUC, is a measure of choice for assessing the performance of a classifier for imbalanced data. AUC maximization refers to a learning paradigm that learns a predictive model by directly maximizing its AUC score. It has been studied for more than two decades dating back to late 90s, and a huge amount of work has been devoted to AUC maximization since then. Recently, stochastic AUC maximization for big data and deep AUC maximization (DAM) for deep learning have received increasing attention and yielded dramatic impact for solving real-world problems. However, to the best our knowledge, there is no comprehensive survey of related works for AUC maximization. This article aims to address the gap by reviewing the literature in the past two decades. We not only give a holistic view of the literature but also present detailed explanations and comparisons of different papers from formulations to algorithms and theoretical guarantees. We also identify and discuss remaining and emerging issues for DAM and provide suggestions on topics for future work.

AUC optimization for deep learning-based voice activity detection

Voice activity detection (VAD) based on deep neural networks (DNN) have demonstrated good performance in adverse acoustic environments. Current DNN-based VAD optimizes a surrogate function, e.g., minimum cross-entropy or minimum squared error, at a given decision threshold. However, VAD usually works on-the-fly with a dynamic decision threshold, and the receiver operating characteristic (ROC) curve is a global evaluation metric for VAD at all possible decision thresholds. In this paper, we propose to maximize the area under the ROC curve (MaxAUC) by DNN, which can maximize the performance of VAD in terms of the entire ROC curve. However, the objective of the AUC maximization is nondifferentiable. To overcome this difficulty, we relax the nondifferentiable loss function to two differentiable approximation functions—sigmoid loss and hinge loss. To study the effectiveness of the proposed MaxAUC-DNN VAD, we take either a standard feedforward neural network or a bidirectional long short-term memory network as the DNN model with either the state-of-the-art multi-resolution cochleagram or short-term Fourier transform as the acoustic feature. We conducted noise-independent training to all comparison methods. Experimental results show that taking AUC as the optimization objective results in higher performance than the common objectives of the minimum squared error and minimum cross-entropy. The experimental conclusion is consistent across different DNN structures, acoustic features, noise scenarios, training sets, and languages.

Pairwise learning problems with regularization networks and Nyström subsampling approach

Pairwise learning usually refers to the learning problem that works with pairs of training samples, such as ranking, similarity and metric learning, and AUC maximization. To overcome the challenge of pairwise learning in the large scale computation, this paper introduces Nyström sampling approach to the coefficient-based regularized pairwise algorithm in the context of kernel networks. Our theorems establish that the obtained Nyström estimator achieves the minimax error over all estimators using the whole data provided that the subsampling level is not too small. We derive the function relation between the subsampling level and regularization parameter that guarantees computation cost reduction and asymptotic behaviors’ optimality simultaneously. The Nyström coefficient-based pairwise learning method does not require the kernel to be symmetric or positive semi-definite, which provides more flexibility and adaptivity in the learning process. We apply the method to the bipartite ranking problem, which improves the state-of-the-art theoretical results in previous works. By developing probability inequalities for U-statistics on Hilbert–Schmidt operators, we provide new mathematical tools for handling pairs of examples involved in pairwise learning.

Balanced Self-Paced Learning for AUC Maximization

Learning to improve AUC performance is an important topic in machine learning. However, AUC maximization algorithms may decrease generalization performance due to the noisy data. Self-paced learning is an effective method for handling noisy data. However, existing self-paced learning methods are limited to pointwise learning, while AUC maximization is a pairwise learning problem. To solve this challenging problem, we innovatively propose a balanced self-paced AUC maximization algorithm (BSPAUC). Specifically, we first provide a statistical objective for self-paced AUC. Based on this, we propose our self-paced AUC maximization formulation, where a novel balanced self-paced regularization term is embedded to ensure that the selected positive and negative samples have proper proportions. Specially, the sub-problem with respect to all weight variables may be non-convex in our formulation, while the one is normally convex in existing self-paced problems. To address this, we propose a doubly cyclic block coordinate descent method. More importantly, we prove that the sub-problem with respect to all weight variables converges to a stationary point on the basis of closed-form solutions, and our BSPAUC converges to a stationary point of our fixed optimization objective under a mild assumption. Considering both the deep learning and kernel-based implementations, experimental results on several large-scale datasets demonstrate that our BSPAUC has a better generalization performance than existing state-of-the-art AUC maximization methods.

Robust pairwise learning with Huber loss

Pairwise learning naturally arises from machine learning tasks such as AUC maximization, ranking, and metric learning. In this paper we propose a new pairwise learning algorithm based on the additive noise regression model, which adopts the pairwise Huber loss and applies effectively even to the situation where the noise only satisfies a weak moment condition. Owing to the robustness of Huber loss function, this new method is resistant to heavy-tailed errors or outliers in the response variable. We establish a comparison theorem to characterize the gap between the excess generalization error and the prediction error. We derive the error bounds and convergence rates under appropriate conditions. It is worth mentioning that all the results are established under the (1+ϵ)-th moment condition of the noise variable. It is rather weak particularly in the case of ϵ<1, which means the noise variable does not even admit a finite variance.

Large-Scale Nonlinear AUC Maximization via Triply Stochastic Gradients.

Learning to improve AUC performance for imbalanced data is an important machine learning research problem. Most methods of AUC maximization assume that the model function is linear in the original feature space. However, this assumption is not suitable for nonlinear separable problems. Although there have been some nonlinear methods of AUC maximization, scaling up nonlinear AUC maximization is still an open question. To address this challenging problem, in this paper, we propose a novel large-scale nonlinear AUC maximization method (named as TSAM) based on the triply stochastic gradient descents. Specifically, we first use the random Fourier feature to approximate the kernel function. After that, we use the triply stochastic gradients w.r.t. the pairwise loss and random feature to iteratively update the solution. Finally, we prove that TSAM converges to the optimal solution with the rate of O(1/t) after t iterations. Experimental results on a variety of benchmark datasets not only confirm the scalability of TSAM, but also show a significant reduction of computational time compared with existing batch learning algorithms, while retaining the similar generalization performance.

Pairwise Learning with Differential Privacy Guarantees

Pairwise learning has received much attention recently as it is more capable of modeling the relative relationship between pairs of samples. Many machine learning tasks can be categorized as pairwise learning, such as AUC maximization and metric learning. Existing techniques for pairwise learning all fail to take into consideration a critical issue in their design, i.e., the protection of sensitive information in the training set. Models learned by such algorithms can implicitly memorize the details of sensitive information, which offers opportunity for malicious parties to infer it from the learned models. To address this challenging issue, in this paper, we propose several differentially private pairwise learning algorithms for both online and offline settings. Specifically, for the online setting, we first introduce a differentially private algorithm (called OnPairStrC) for strongly convex loss functions. Then, we extend this algorithm to general convex loss functions and give another differentially private algorithm (called OnPairC). For the offline setting, we also present two differentially private algorithms (called OffPairStrC and OffPairC) for strongly and general convex loss functions, respectively. These proposed algorithms can not only learn the model effectively from the data but also provide strong privacy protection guarantee for sensitive information in the training set. Extensive experiments on real-world datasets are conducted to evaluate the proposed algorithms and the experimental results support our theoretical analysis.

MBA: Mini-Batch AUC Optimization.

Area under the receiver operating characteristics curve (AUC) is an important metric for a wide range of machine-learning problems, and scalable methods for optimizing AUC have recently been proposed. However, handling very large data sets remains an open challenge for this problem. This article proposes a novel approach to AUC maximization based on sampling mini-batches of positive/negative instance pairs and computing U-statistics to approximate a global risk minimization problem. The resulting algorithm is simple, fast, and learning-rate free. We show that the number of samples required for good performance is independent of the number of pairs available, which is a quadratic function of the positive and negative instances. Extensive experiments show the practical utility of the proposed method.

Stochastic AUC optimization with general loss

Recently, there is considerable work on developing efficient stochastic optimization algorithms for AUC maximization. However, most of them focus on the least square loss which may be not the best option in practice. The main difficulty for dealing with the general convex loss is the pairwise nonlinearity w.r.t. the sampling distribution generating the data. In this paper, we use Bernstein polynomials to uniformly approximate the general losses which are able to decouple the pairwise nonlinearity. In particular, we show that this reduction for AUC maximization with a general loss is equivalent to a weakly convex (nonconvex) min-max formulation. Then, we develop a novel SGD algorithm for AUC maximization with per-iteration cost linearly w.r.t. the data dimension, making it amenable for streaming data analysis. Despite its non-convexity, we prove its global convergence by exploring the appealing convexity-preserving property of Bernstein polynomials and the intrinsic structure of the min-max formulation. Experiments are performed to validate the effectiveness of the proposed approach.

Hierarchical detection of abnormal behaviors in video surveillance through modeling normal behaviors based on AUC maximization

Detection of abnormal behaviors is a challenging issue as there are disagreements over how abnormal behaviors should be defined. In this paper, a new method is proposed to estimate a model of normal behaviors and consequently to detect abnormal behaviors. Estimating a model of normal behaviors constitutes a main part of the training phase in detecting abnormal behaviors and is one of the main issues in the field. To estimate such a model, the histogram of the oriented optical flow (HOF) is first extracted as a local-based feature, and then, the proposed features of speed, variance of the trajectory and deviation from the trajectory are extracted as object-based features. Spectral clustering is used to cluster behavioral features that are similar. To design a classifier for estimating the model of normal behaviors in each cluster, a new method is proposed based on maximization of the area under the receiver operating characteristic curve (AUC). This new method does not need manual labeling for estimation of the model, and then, it carries out the estimation process in a semi-supervised fashion. In this paper, a method for hierarchical detection of abnormal behaviors based on the priority of the behavioral features is also proposed in the testing phase of the study. To this purpose, the existence of an abnormal behavior is hierarchically detected through using the HOF feature and the three object-based features of speed, variance of the trajectory and deviation from the trajectory. The distinguishing characteristic of the proposed method is that, as soon as the abnormal behavior is detected along the hierarchical application of the features, the process stops. The experimental results show that the proposed method can effectively detect the abnormal behaviors in several databases and achieve comparable performance to the state-of-the-art methods for detection of abnormal behaviors.

Scalable and Efficient Pairwise Learning to Achieve Statistical Accuracy

Pairwise learning is an important learning topic in the machine learning community, where the loss function involves pairs of samples (e.g., AUC maximization and metric learning). Existing pairwise learning algorithms do not perform well in the generality, scalability and efficiency simultaneously. To address these challenging problems, in this paper, we first analyze the relationship between the statistical accuracy and the regularized empire risk for pairwise loss. Based on the relationship, we propose a scalable and efficient adaptive doubly stochastic gradient algorithm (AdaDSG) for generalized regularized pairwise learning problems. More importantly, we prove that the overall computational cost of AdaDSG is O(n) to achieve the statistical accuracy on the full training set with the size of n, which is the best theoretical result for pairwise learning to the best of our knowledge. The experimental results on a variety of real-world datasets not only confirm the effectiveness of our AdaDSG algorithm, but also show that AdaDSG has significantly better scalability and efficiency than the existing pairwise learning algorithms.