Leader learning loss function in neural network classification

Abstract

Deep learning, based on Empirical Risk Minimization (ERM), typically aims to fit the ideal outputs of all samples due to its large capacity. However, models trained based on empirical losses like cross entropy (CE) or mean square error (MSE), often learn unnecessary information during classification, leading to premature overfitting. On the other hand, the result-focused loss functions, i.e., zero–one loss or hinge loss, are hard to optimize and thus are rarely applied directly in neural network. This paper proposes a novel leader learning in classification, where CE is gradually trained by classification results using sample-dependent cost-sensitive learning. As complementary, the stepwise-changed CE covers the deficiency on classification error while preserving the advantage of fast convergence. In this way, the deviation between CE and classification error can be corrected. Experimental results demonstrate that the proposed leader learning has a more significant convergence trend than the baseline algorithms. Moreover, the loss function learned from a specific dataset has broad generality that can be transferred to other models as prior knowledge.