Category-aware feature attribution for Self-Optimizing medical image classification

Abstract

Convolutional Neural Networks (CNNs) have achieved state-of-the-art performance for automatic medical image classification. Although remarkable advancement has been made, they have not demonstrated sufficiently accurate and robust results for clinical use. In addition, they are limited by the unavoidable hard samples and minority class samples. Meanwhile, it is hard to understand the prediction mechanism and debug the wrong prediction. To address these issues, this paper proposes a Self-Optimizing medical image classification framework based on category-aware feature attribution, which automatically performs model diagnosis, repairing, and enhancement. On one hand, we introduce selective filter restraining to alleviate the disturbance of misclassified features for hard samples. On the other hand, we introduce sample-specific fine-tuning to promote category-wise feature learning, especially for minority classes. The Self-Optimizing framework is a versatile module for other medical image classifiers based on mainstream CNN structures. We extensively validated the proposed method on three typical yet challenging medical image classification tasks: (i) skin lesion classification from dermoscopic images in the International Skin Imaging Collaboration (ISIC) 2019 dataset, (ii) breast cancer classification from ultrasound images in Breast Ultrasound Images (BUSI) dataset, and (iii) breast tumour classification from histopathologic scans of lymph node sections in PatchCamelyon (PCam) dataset. Experimental results demonstrate that the proposed framework achieves 1%∼5% performance gain for various classifiers, brings better explainability, and can be applied to incremental learning scenarios.