Surrogate Loss Research Articles

FairDRO: Group fairness regularization via classwise robust optimization

Existing group fairness-aware training methods fall into two categories: re-weighting underrepresented groups according to certain rules, or using regularization terms such as smoothed approximations of fairness metrics or surrogate statistical quantities. While each category has its own strength in applicability or performance when compared to each other, their successful performances are typically limited to specific cases. To that end, we propose a new approach called FairDRO, which takes advantage of both categories through a classwise group distributionally robust optimization (DRO) framework. Our method unifies re-weighting and regularization by incorporating a well-justified group fairness metric into the objective as regularization, but solving it through a principled re-weighting strategy. To optimize our resulting objective efficiently, we adopt an iterative algorithm and consequently develop two variants of FairDRO algorithm depending on the choice of surrogate loss. For in-depth understanding, we derive three theoretical results: (i) a closed-form solution for the correct re-weights; (ii) justifications for using the surrogate losses; and (iii) a convergence analysis of our method. Experimental results show that our algorithms consistently achieve state-of-the-art performance in accuracy-fairness trade-offs across multiple benchmarks, demonstrating scalability and broad applicability compared to existing methods.

Online imbalance learning with unpredictable feature evolution and label scarcity

Recently, online learning with imbalanced data streams has aroused wide concern, which reflects an uneven distribution of different classes in data streams. Existing approaches have conventionally been conducted on stationary feature space and they assume that we can obtain the entire labels of data streams in the case of supervised learning. However, in many real scenarios, e.g., the environment monitoring task, new features flood in, and old features are partially lost during the changing environment as the different lifespans of different sensors. Besides, each instance needs to be labeled by experts, resulting in expensive costs and scarcity of labels. To address the above problems, this paper proposes a novel Online Imbalance learning with unpredictable Feature evolution and Label scarcity (OIFL) algorithm. First, we utilize margin-based online active learning to selectively label valuable instances. After obtaining the labels, we handle imbalanced class distribution by optimizing F-measure and transforming F-measure optimization into a weighted surrogate loss minimization. When data streams arrive with augmented features, we combine the online passive-aggressive algorithm and structural risk minimization to update the classifier in the divided feature space. When data streams arrive with incomplete features, we leverage variance to identify the most informative features following the empirical risk minimization principle and continue to update the existing classifier as before. Finally, we obtain a sparse but reliable learner by the strategy of projecting truncation. We derive theoretical analyses of OIFL. Also, experiments on the synthetic datasets and real-world data streams to validate the effectiveness of our method.

Communication‐Efficient Distributed Estimation of Causal Effects With High‐Dimensional Data

ABSTRACTWe propose a communication‐efficient algorithm to estimate the average treatment effect (ATE), when the data are distributed across multiple sites and the number of covariates is possibly much larger than the sample size in each site. Our main idea is to calibrate the estimates of the propensity score and outcome models using some proper surrogate loss functions to approximately attain the desired covariate balancing property. We show that under possible model misspecification, our distributed covariate balancing propensity score estimator (disthdCBPS) can approximate the global estimator, obtained by pooling together the data from multiple sites, at a fast rate. Thus, our estimator remains consistent and asymptotically normal. In addition, when both the propensity score and the outcome models are correctly specified, the proposed estimator attains the semi‐parametric efficiency bound. We illustrate the empirical performance of the proposed method in both simulation and empirical studies.

IRDA: Implicit data augmentation for deep imbalanced regression

Imbalanced data distributions are prevalent in real-world classification and regression tasks. Data augmentation is a commonly employed technique to mitigate this issue, with implicit methods gaining attention for their effectiveness and efficiency. However, implicit data augmentation methods have not been extensively explored in the context of regression tasks. To address this gap, we introduce IRDA, a novel learning method for regression that incorporates implicit data augmentation. Our approach includes developing a new augmentation strategy specifically tailored for deep imbalanced regression tasks, and a regression loss function that is suitable for real-world data with imbalanced label distributions and non-uniformly distributed features. We derive an easily computable surrogate loss and propose two implicit data augmentation algorithms, one incorporating meta-learning and one without. Additionally, we provide a regularization perspective to offer a deeper understanding of IRDA. We evaluate IRDA on five datasets, including a large-scale dataset, demonstrating its effectiveness in mitigating the adverse effects of imbalanced data distribution and its adaptability to various regression tasks.

A model-free variable screening method for optimal treatment regimes with high-dimensional survival data

Summary We propose a model-free variable screening method for the optimal treatment regime with high-dimensional survival data. The proposed screening method provides a unified framework to select the active variables in a prespecified target population, including the treated group as a special case. Based on this framework, the optimal treatment regime is exactly the optimal classifier that minimizes a weighted misclassification error rate, with weights associated with survival outcome variables, the censoring distribution and a prespecified target population. Our main contribution involves reformulating the weighted classification problem into a classification problem within a hypothetical population, where the observed data can be viewed as a sample obtained from outcome-dependent sampling, with the selection probability inversely proportional to the weights. Consequently, we introduce the weighted Kolmogorov–Smirnov approach for selecting active variables in the optimal treatment regime, extending the conventional Kolmogorov–Smirnov method for binary classification. Additionally, the proposed screening method exhibits two levels of robustness. The first level of robustness is achieved because the proposed method does not require any model assumptions for the survival outcome on treatment and covariates, whereas the other is attained as the form of treatment regimes is allowed to be unspecified even without requiring convex surrogate loss, such as logit loss or hinge loss. As a result, the proposed screening method is robust to model misspecifications, and nonparametric learning methods such as random forests and boosting can be applied to those selected variables for further analysis. The theoretical properties of the proposed method are established. The performance of the proposed method is examined through simulation studies and illustrated by a lung cancer dataset.

A Convergence Path to Deep Learning on Noisy Labels.

In many real-world machine learning classification applications, the model performance based on deep neural networks (DNNs) oftentimes suffers from label noise. Various methods have been proposed in the literature to address this issue, primarily by focusing on designing noise-tolerant loss functions, cleaning label noise, and correcting the objective loss. However, the noise-tolerant loss functions face challenges when the noise level increases. This article aims to reveal a convergence path of a trained model in the presence of label noise, and here, the convergence path depicts the evolution of a trained model over epochs. We first propose a theorem to demonstrate that any surrogate loss function can be used to learn DNNs from noisy labels. Next, theories on the general convergence path for the deep models under label noise are presented and verified through a series of experiments. In addition, we design an algorithm based on the proposed theorems that make efficient corrections on the noisy labels and achieve strong robustness in the DNN models. We designed several experiments using benchmark datasets to assess noise tolerance and verify the theorems presented in this article. The comprehensive experimental results firmly confirm our theoretical results and also clearly validate the effectiveness of our method under various levels of label noise.

Finding the optimal dynamic treatment regimes using smooth Fisher consistent surrogate loss

Finding the optimal dynamic treatment regimes using smooth Fisher consistent surrogate loss

L1-penalized AUC-optimization with a surrogate loss

L1-penalized AUC-optimization with a surrogate loss

Simultaneous Optimization of Bid Shading and Internal Auction for Demand-Side Platforms

Online advertising has been one of the most important sources for industry's growth, where the demand-side platforms (DSP) play an important role via bidding to the ad exchanges on behalf of their advertiser clients. Since more and more ad exchanges have shifted from second to first price auctions, it is challenging for DSPs to adjust bidding strategy in the volatile environment. Recent studies on bid shading in first-price auctions may have limited performance due to relatively strong hypotheses about winning probability distribution. Moreover, these studies do not consider the incentive of advertiser clients, which can be crucial for a reliable advertising platform. In this work, we consider both the optimization of bid shading technique and the design of internal auction which is ex-post incentive compatible (IC) for the management of a DSP. Firstly, we prove that the joint design of bid shading and ex-post IC auction can be reduced to choosing one monotone bid function for each advertiser without loss of optimality. Then we propose a parameterized neural network to implement the monotone bid functions. With well-designed surrogate loss, the objective can be optimized in an end-to-end manner. Finally, our experimental results demonstrate the effectiveness and superiority of our algorithm.

Perturbation-Invariant Adversarial Training for Neural Ranking Models: Improving the Effectiveness-Robustness Trade-Off

Neural ranking models (NRMs) have shown great success in information retrieval (IR). But their predictions can easily be manipulated using adversarial examples, which are crafted by adding imperceptible perturbations to legitimate documents. This vulnerability raises significant concerns about their reliability and hinders the widespread deployment of NRMs. By incorporating adversarial examples into training data, adversarial training has become the de facto defense approach to adversarial attacks against NRMs. However, this defense mechanism is subject to a trade-off between effectiveness and adversarial robustness. In this study, we establish theoretical guarantees regarding the effectiveness-robustness trade-off in NRMs. We decompose the robust ranking error into two components, i.e., a natural ranking error for effectiveness evaluation and a boundary ranking error for assessing adversarial robustness. Then, we define the perturbation invariance of a ranking model and prove it to be a differentiable upper bound on the boundary ranking error for attainable computation. Informed by our theoretical analysis, we design a novel perturbation-invariant adversarial training (PIAT) method for ranking models to achieve a better effectiveness-robustness trade-off. We design a regularized surrogate loss, in which one term encourages the effectiveness to be maximized while the regularization term encourages the output to be smooth, so as to improve adversarial robustness. Experimental results on several ranking models demonstrate the superiority of PITA compared to existing adversarial defenses.

Variable selection of Kolmogorov-Smirnov maximization with a penalized surrogate loss

Kolmogorov-Smirnov (KS) statistic is quite popular in many areas as the major performance evaluation criterion for binary classification due to its explicit business intension. Fang and Chen (2019) proposed a novel DMKS method that directly maximizes the KS statistic and compares favorably with the popular existing methods. However, DMKS did not consider the critical problem of variable selection since the special form of KS brings great challenge to establish the DMKS estimator's asymptotic distribution which is most likely to be nonstandard. This intractable issue is handled by introducing a surrogate loss function which leads to a n-consistent estimator for the true parameter up to a multiplicative scalar. Then a nonconcave penalty function is combined to achieve the variable selection consistency and asymptotical normality with the oracle property. Results of empirical studies confirm the theoretical results and show advantages of the proposed SKS (Surrogated Kolmogorov-Smirnov) method compared to the original DMKS method without variable selection.

Communication-Efficient Nonparametric Quantile Regression via Random Features

This article introduces a refined algorithm designed for distributed nonparametric quantile regression in a reproducing kernel Hilbert space (RKHS). Unlike existing nonparametric approaches that primarily address homogeneous data, our approach uses kernel-based quantile regression to effectively model heterogeneous data. Moreover, we integrate the concepts of random features (RF) and communication-efficient surrogate likelihood (CSL) to ensure accurate estimation and enhance computational efficiency in distributed settings. Specifically, we employ an embedding technique to map the original data into RF spaces, enabling us to construct an extended surrogate loss function. This function can be locally optimized using an iterative alternating direction method of multipliers (ADMM) algorithm, minimizing the need for extensive computation and communication within the distributed system. The article thoroughly investigates the asymptotic properties of the distributed estimator and provides convergence rates of the excess risk. These properties are established under mild technical conditions and are comparable to state-of-the-art results in the literature. Additionally, we validate the effectiveness of the proposed algorithm through a comprehensive set of synthetic examples and a real data study, effectively highlighting its advantages and practical utility. Supplementary materials for this article are available online.

Certified Distributional Robustness on Smoothed Classifiers

The robustness of deep neural networks (DNNs) against adversarial example attacks has raised wide attention. For smoothed classifiers, we propose the worst-case adversarial loss over input distributions as a robustness certificate. Compared with previous certificates, our certificate better describes the empirical performance of the smoothed classifiers. By exploiting duality and the smoothness property, we provide an easy-to-compute upper bound as a surrogate for the certificate. We adopt a noisy adversarial learning procedure to minimize the surrogate loss to improve model robustness. We show that our training method provides a theoretically tighter bound over the distributional robust base classifiers. Experiments on a variety of datasets further demonstrate superior robustness performance of our method over the state-of-the-art certified or heuristic methods.

EVAE: Evolutionary Variational Autoencoder.

Variational autoencoders (VAEs) are challenged by the imbalance between representation inference and task fitting caused by surrogate loss. To address this issue, existing methods adjust their balance by directly tuning their coefficients. However, these methods suffer from a tradeoff uncertainty, i.e., nondynamic regulation over iterations and inflexible hyperparameters for learning tasks. Accordingly, we make the first attempt to introduce an evolutionary VAE (eVAE), building on the variational information bottleneck (VIB) theory and integrative evolutionary neural learning. eVAE integrates a variational genetic algorithm (VGA) into VAE with variational evolutionary operators, including variational mutation (V-mutation), crossover, and evolution. Its training mechanism synergistically and dynamically addresses and updates the learning tradeoff uncertainty in the evidence lower bound (ELBO) without additional constraints and hyperparameter tuning. Furthermore, eVAE presents an evolutionary paradigm to tune critical factors of VAEs and addresses the premature convergence and random search problem in integrating evolutionary optimization into deep learning. Experiments show that eVAE addresses the KL-vanishing problem for text generation with low reconstruction loss, generates all the disentangled factors with sharp images, and improves image generation quality. eVAE achieves better disentanglement, generation performance, and generation-inference balance than its competitors. Code available at: https://github.com/amasawa/eVAE.

An asymptotically normal representation for the minimal clinically important difference under a nonconvex surrogate loss

An asymptotically normal representation for the minimal clinically important difference under a nonconvex surrogate loss

Noisy Label Learning With Provable Consistency for a Wider Family of Losses.

Deep models have achieved state-of-the-art performance on a broad range of visual recognition tasks. Nevertheless, the generalization ability of deep models is seriously affected by noisy labels. Though deep learning packages have different losses, this is not transparent for users to choose consistent losses. This paper addresses the problem of how to use abundant loss functions designed for the traditional classification problem in the presence of label noise. We present a dynamic label learning (DLL) algorithm for noisy label learning and then prove that any surrogate loss function can be used for classification with noisy labels by using our proposed algorithm, with a consistency guarantee that the label noise does not ultimately hinder the search for the optimal classifier of the noise-free sample. In addition, we provide a depth theoretical analysis of our algorithm to verify the justifies' correctness and explain the powerful robustness. Finally, experimental results on synthetic and real datasets confirm the efficiency of our algorithm and the correctness of our justifies and show that our proposed algorithm significantly outperforms or is comparable to current state-of-the-art counterparts.

Towards Building a Trustworthy Deep Learning Framework for Medical Image Analysis.

Computer vision and deep learning have the potential to improve medical artificial intelligence (AI) by assisting in diagnosis, prediction, and prognosis. However, the application of deep learning to medical image analysis is challenging due to limited data availability and imbalanced data. While model performance is undoubtedly essential for medical image analysis, model trust is equally important. To address these challenges, we propose TRUDLMIA, a trustworthy deep learning framework for medical image analysis, which leverages image features learned through self-supervised learning and utilizes a novel surrogate loss function to build trustworthy models with optimal performance. The framework is validated on three benchmark data sets for detecting pneumonia, COVID-19, and melanoma, and the created models prove to be highly competitive, even outperforming those designed specifically for the tasks. Furthermore, we conduct ablation studies, cross-validation, and result visualization and demonstrate the contribution of proposed modules to both model performance (up to 21%) and model trust (up to 5%). We expect that the proposed framework will support researchers and clinicians in advancing the use of deep learning for dealing with public health crises, improving patient outcomes, increasing diagnostic accuracy, and enhancing the overall quality of healthcare delivery.

Optimizing Two-way Partial AUC with an End-to-end Framework.

The Area Under the ROC Curve (AUC) is a crucial metric for machine learning, which evaluates the average performance over all possible True Positive Rates (TPRs) and False Positive Rates (FPRs). Based on the knowledge that a skillful classifier should simultaneously embrace a high TPR and a low FPR, we turn to study a more general variant called Two-way Partial AUC (TPAUC), where only the region with TPR ≥ α, FPR ≤ β is included in the area. Moreover, a recent work shows that the TPAUC is essentially inconsistent with the existing Partial AUC metrics where only the FPR range is restricted, opening a new problem to seek solutions to leverage high TPAUC. Motivated by this, we present the first trial in this paper to optimize this new metric. The critical challenge along this course lies in the difficulty of performing gradient-based optimization with end-to-end stochastic training, even with a proper choice of surrogate loss. To address this issue, we propose a generic framework to construct surrogate optimization problems, which supports efficient end-to-end training with deep learning. Moreover, our theoretical analyses show that: 1) the objective function of the surrogate problems will achieve an upper bound of the original problem under mild conditions, and 2) optimizing the surrogate problems leads to good generalization performance in terms of TPAUC with a high probability. Finally, empirical studies over several benchmark datasets speak to the efficacy of our framework.

Prescriptive analytics for a maritime routing problem

Port state control (PSC) serves as the final defense against substandard ships in maritime transportation. The port state control officer (PSCO) routing problem involves selecting ships for inspection and determining the inspection sequence for available PSCOs, aiming to identify the highest number of deficiencies. Port authorities face this problem daily, making decisions without prior knowledge of ship conditions. Traditionally, a predict-then-optimize framework is employed, but its machine learning (ML) models' loss function fails to account for the impact of predictions on the downstream optimization problem, potentially resulting in suboptimal decisions. We adopt a decision-focused learning framework, integrating the PSCO routing problem into the ML models' training process. However, as the PSCO routing problem is NP-hard and plugging it into the training process of ML models requires that it be solved numerous times, computational complexity and scalability present significant challenges. To address these issues, we first convert the PSCO routing problem into a compact model using undominated inspection templates, enhancing the model's solution efficiency. Next, we employ a family of surrogate loss functions based on noise-contrastive estimation (NCE) for the ML model, requiring a solution pool treating suboptimal solutions as noise samples. This pool represents a convex hull of feasible solutions, avoiding frequent reoptimizations during the ML model's training process. Through computational experiments, we compare the predictive and prescriptive qualities of both the two-stage framework and the decision-focused learning framework under varying instance sizes. Our findings suggest that accurate predictions do not guarantee good decisions; the decision-focused learning framework's performance may depend on the optimization problem size and the training dataset size; and using a solution pool containing noise samples strikes a balance between training efficiency and decision performance.

Communication-efficient estimation of quantile matrix regression for massive datasets

In modern scientific applications, more and more data sets contain natural matrix predictors and traditional regression methods are not directly applicable. Matrix regression has been adapted to such data structure and received increasing attention in recent years. In this paper, we consider estimation of the conditional quantile in high-dimensional regularized matrix regression with a nuclear norm penalty and establish the convergence rate of the estimator. In order to construct a quantile matrix regression estimator in the distributed setting or for massive data sets, we propose a regularized communication-efficient surrogate loss (CSL) function. The proposed CSL method only needs the worker machines to compute the gradient based on local data and the central machine solves a regularized estimation problem. We prove that the estimation error based on the proposed CSL method matches the estimation error bound of the centralized method that analyzes the entire data set. An alternating direction method of multipliers algorithm is developed to efficiently obtain the distributed CSL estimator. The finite-sample performance of the proposed estimator is studied through simulations and an application to Beijing Air Quality data set.