RBUE: a ReLU-based uncertainty estimation method for convolutional neural networks

Abstract

Convolutional neural networks (CNNs) have successfully demonstrated their powerful predictive performance in a variety of tasks. However, it remains a challenge to estimate the uncertainty of these predictions simply and accurately. Deep Ensemble is widely considered the state-of-the-art method which can estimate the uncertainty accurately, but it is expensive to train and test. MC-Dropout is another popular method that is less costly but lacks the diversity of predictions resulting in less accurate uncertainty estimates. To combine the benefits of both, we introduce a ReLU-Based Uncertainty Estimation (RBUE) method. Instead of using the randomness of the Dropout module during the test phase (MC-Dropout) or using the randomness of the initial weights of CNNs (Deep Ensemble), RBUE uses the randomness of activation function to obtain diverse outputs in the testing phase to estimate uncertainty. Under the method, we propose strategy MC-DropReLU and develop strategy MC-RReLU. The uniform distribution of the activation function’s position in CNNs allows the randomness to be well transferred to the output results and gives a more diverse output, thus improving the accuracy of the uncertainty estimation. Moreover, our method is simple to implement and does not need to modify the existing model. We experimentally validate the RBUE on three widely used datasets, CIFAR10, CIFAR100, and TinyImageNet. The experiments demonstrate that our method has competitive performance but is more favorable in training time.