Probabilistic Target Encoding of Neural Networks with Softmax Output Nodes

Abstract

When training neural networks with softmax outputs, one-hot target encoding is commonly used due to its simplicity. This strategy does not incorporate the probability that an input sample belongs to a certain class but adopts “one” or “zero” as the desired output values of neural networks. Instead of the most prevalent one-hot encoding, this paper proposes a probabilistic target encoding to prevent the overfitting of neural networks to training samples. This effect brings about the accuracy improvement of test samples. We demonstrated the effectiveness of the proposed target encoding through simulations of multi-layer perceptrons and convolutional neural networks for various classification problems such as handwritten-digit recognition, isolated-word recognition, image classification, and object recognition tasks. The simulation results show that the proposed probabilistic target encoding is superior to one-hot encoding as it prevents the overfitting of neural networks to training samples.