SAM-IE: SAM-based image enhancement for facilitating medical image diagnosis with segmentation foundation model

Abstract

The Segment Anything Model (SAM) is a large-scale model developed for general segmentation tasks in computer vision. Trained on a substantial dataset, SAM can accurately segment various objects in natural scene images. However, due to significant semantic differences between medical and natural images, directly applying SAM to medical image segmentation does not yield optimal results. Therefore, effectively utilizing such a comprehensive foundation model for medical image analysis is an emerging research topic. Despite SAM’s current suboptimal performance in medical image segmentation, it shows preliminary recognition and localization of tissues and lesions that radiologists focus on in medical images. This implies that SAM’s generated masks, features, and stability scores hold potential value for medical image diagnosis. Therefore, based on the model output of SAM, this study introduces a SAM-based Image Enhancement (SAM-IE) method for disease diagnosis. Targeting popular medical image classification models (e.g., ResNet50 and Swin Transformer), SAM-IE is proposed to enhance image inputs by combining the binary mask and contour mask generated by SAM with the original image to create attention maps, thereby improving diagnostic performance. To validate the effectiveness of SAM-IE for diagnosis, experiments were conducted on four medical image datasets for eight classification tasks. The results demonstrate the effectiveness of our proposed SAM-IE model, showcasing SAM’s potential value in medical image classification. This study provides a feasible approach for integrating SAM into disease diagnosis.