Abstract
We derive the fast convergence rates of a deep neural network (DNN) classifier with the rectified linear unit (ReLU) activation function learned using the hinge loss. We consider three cases for a true model: (1) a smooth decision boundary, (2) smooth conditional class probability, and (3) the margin condition (i.e., the probability of inputs near the decision boundary is small). We show that the DNN classifier learned using the hinge loss achieves fast rate convergences for all three cases provided that the architecture (i.e., the number of layers, number of nodes and sparsity) is carefully selected. An important implication is that DNN architectures are very flexible for use in various cases without much modification. In addition, we consider a DNN classifier learned by minimizing the cross-entropy, and show that the DNN classifier achieves a fast convergence rate under the conditions that the noise exponent and margin exponent are large. Even though they are strong, we explain that these two conditions are not too absurd for image classification problems. To confirm our theoretical explanation, we present the results of a small numerical study conducted to compare the hinge loss and cross-entropy.
Highlights
Deep learning [Hinton and Salakhutdinov, 2006, Larochelle et al, 2007, Goodfellow et al, 2016] has received much attention for dimension reduction and classification of objects, such as images, speech, and language
Many researchers have demonstrated that deep neural networks (DNNs) are much more efficient in representing certain complex functions than their shallow counterparts [Montufar et al, 2014, Raghu et al, 2016, Eldan and Shamir, 2016], which has been reconfirmed by Yarotsky [2017] and Petersen and Voigtlaender [2018], who showed that DNNs can approximate a large class of functions, including even discontinuous functions with a parsimonious number of parameters
We prove that the estimation of a classifier based on the DNN with the hinge loss can achieve fast convergence rates under various situations
Summary
Deep learning [Hinton and Salakhutdinov, 2006, Larochelle et al, 2007, Goodfellow et al, 2016] has received much attention for dimension reduction and classification of objects, such as images, speech, and language. We justify the use of the cross-entropy in learning a DNN by showing that the corresponding classifier achieves a fast convergence rate when most data have a conditional class probability close to 1 or zero. Note that this assumption is reasonable for image recognition because human beings recognize most real world images quite well.
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have