Mitigating data imbalance and noise: A divergence-based approach with enhanced sample selection

Abstract

Label noise and distribution imbalance are common challenges in real-world datasets. To address noisy labels, sample selection methods are typically used. These methods identify small-loss data as correctly labeled and utilize only these clean samples for updating parameters during training. However, our thorough analysis shows that in scenarios with a long-tailed distribution, methods based on small-loss tend to favor samples from dominant classes, leading to a significant drop in performance. To tackle this issue, we introduce a divergence-based separation mechanism coupled with prototype distance for clean sample identification across each class, effectively mitigating the adverse effects of distribution imbalance on sample selection. Furthermore, drawing on recent theoretical insights that highlight the benefits of pseudo-labels in managing class imbalances, we enhance our approach by integrating semi-supervised and contrastive learning techniques. Comprehensive tests conducted on two synthetic noisy long-tailed datasets and one real-world dataset demonstrate the efficacy of our method in dealing with label noise and long-tailed distributions.