Analyzing Deep Neural Networks with Noisy Labels

Abstract

Training deep neural networks (DNNs) with noisy labels is a fundamental problem in achieving generalization in DNNs. Recent studies mainly adopt sample selection in which the samples with small losses are regarded as clean ones, based on the findings that DNNs tend to learn simple and easy patterns first, and then gradually memorize all data. In this paper, we further investigate the sample selection by observing the loss distribution of training samples with clean and noisy labels. In experimental results, we found that the loss distributions with clean and noisy samples are different at the early stage, and gradually converge into similar distributions. Besides, this phenomenon is notably different depending on the datasets and noise types, such as symmetric and pair. Based on these findings, we argue that the sample selection method should consider an early stopping condition in learning DNNs with noisy labels.