Data valuation for medical imaging using Shapley value and application to a large-scale chest X-ray dataset

Siyi Tang,Amirata Ghorbani,Sameer Rehman,Jared A Dunnmon,James Zou,Daniel L Rubin,Rikiya Yamashita

doi:10.1038/s41598-021-87762-2

Abstract

The reliability of machine learning models can be compromised when trained on low quality data. Many large-scale medical imaging datasets contain low quality labels extracted from sources such as medical reports. Moreover, images within a dataset may have heterogeneous quality due to artifacts and biases arising from equipment or measurement errors. Therefore, algorithms that can automatically identify low quality data are highly desired. In this study, we used data Shapley, a data valuation metric, to quantify the value of training data to the performance of a pneumonia detection algorithm in a large chest X-ray dataset. We characterized the effectiveness of data Shapley in identifying low quality versus valuable data for pneumonia detection. We found that removing training data with high Shapley values decreased the pneumonia detection performance, whereas removing data with low Shapley values improved the model performance. Furthermore, there were more mislabeled examples in low Shapley value data and more true pneumonia cases in high Shapley value data. Our results suggest that low Shapley value indicates mislabeled or poor quality images, whereas high Shapley value indicates data that are valuable for pneumonia detection. Our method can serve as a framework for using data Shapley to denoise large-scale medical imaging datasets.

Highlights

The reliability of machine learning models can be compromised when trained on low quality data
We aim to characterize the effectiveness of data Shapley in identifying low quality and valuable data in ChestX-ray[14], a large public chest X-ray dataset whose pathology labels were extracted from X-ray reports using text mining techniques
In addition to low value data, our method informs us of data points that are valuable to pneumonia detection

Summary

Introduction

The reliability of machine learning models can be compromised when trained on low quality data. We used data Shapley, a data valuation metric, to quantify the value of training data to the performance of a pneumonia detection algorithm in a large chest X-ray dataset. Modern machine learning methods such as deep learning have achieved impressive empirical performance in diverse medical image analysis tasks, including assessment of cardiac function from ultrasound[1], segmentation of neuronal structures from electron microscopy[2], skin lesion classification from dermatoscopy[3], intracranial hemorrhage detection from computed tomography[4,5], and automated chest X-ray interpretations[6,7] These successes were generally made possible by the availability of large-scale hand-labeled training datasets. We aim to assess the effectiveness of data Shapley in capturing low quality data as well as informing valuable data in the context of pneumonia detection from chest X-ray images

Objectives

Methods

Results

Conclusion