Fidelity: A Property of Deep Neural Networks to Measure the Trustworthiness of Prediction Results

Abstract

With the increasing performance of deep learning on many security-critical tasks, such as face recognition and malware detection, the security issues of machine learning (ML) have become increasingly prominent. Recent studies have shown that deep learning is vulnerable to adversarial examples: carefully crafted inputs with negligible perturbations on legitimate samples could mislead a deep neural network (DNN) to produce adversary-selected outputs while humans can still correctly classify them. Therefore, we need an additional measurement on the trustworthiness of the results of a machine learning model, especially in adversarial settings. In this paper, we analyse the root cause of adversarial examples, and propose a new property, namely fidelity, of machine learning models to describe the gap between what a model learns and the ground truth learned by humans. One of its benefits is detecting adversarial attacks. We formally define fidelity, and propose a novel approach to quantify it. We evaluate the quantification of fidelity of DNNs in adversarial settings on two neural networks. Our preliminary results show that involving the fidelity enables a DNN system to detect adversarial examples with true positive rate 97.7%, and false positive rate 1.67% on a studied DNN model.