460 The role of data augmentation on the performance of automated lesion classification in the presence of imaging artifacts: An evaluation of the 2019 ISIC Challenge

Abstract

Convolutional Neural Networks (CNNs) have been shown to achieve dermatologist level accuracy for skin lesion classification, but models have been limited by size and diversity of training datasets. Data augmentation adds alterations (e.g., blur, flipping, etc.) to images to increase classifier robustness to real-world perturbations and artifacts. We identified the augmentation techniques used in 123 submissions to the International Skin Imaging Collaboration (ISIC) 2019 Grand Challenge and compared each classifier’s performance on images with and without artifacts. We developed a multiclass-multilabel CNN trained to detect artifacts seen in clinic: hair, blur, rulers, and pen markings, and achieved Area Under the Curve (AUC) of >90% for all artifacts except blur (83%). We assessed ISIC 2019 submissions’ performances using balanced multiclass accuracy. Hair and pen markings resulted in a decreased accuracy in 72% and 66% of algorithms respectively, while ruler markings increased the accuracy in 70%. No specific augmentation technique was tied to improved or diminished performance on images with hair, ruler, or pen. 16/123 algorithms used artificial blur and had a 26% better performance than those that did not (p=0.0014). The top 5 algorithms had an average of 8.2 augmentation techniques (59% accuracy) compared to 1.4 in the bottom 5 (18% accuracy), supporting that data augmentation is vital for performance. Our work will shape the development of classifiers for melanoma diagnosis. We introduce a novel artifact classifier useful for quality assurance of dermoscopic images. We show that the diagnostic performance of algorithms on images with hair, ruler, and pen is unaffected by augmentation techniques used, whereas blur is. Machine learning in dermatology may require data augmentation mirroring artifacts seen in clinic, such as artificial hair and pen marker generation. These results will improve the real-world applicability of automated dermatologic classifiers.