Leveraging CNN and Vision Transformer with Transfer Learning to Diagnose Pigmented Skin Lesions

Abstract

The transformer has emerged in computer vision, but whether it can rival or even replace the status of a convolutional neural network (CNN) still lacks deeper cognition. Therefore, this paper will study the feasibility and effectiveness of transformers in identifying the types of pigmented skin lesions by comparing one of the most widely known CNN architectures with three state-of-the-art Vision Transformer (ViT) models. The research results found that the transformer surprisingly outperforms the CNN. In other words, the overall accuracy and F1-score per class of its classifier are much higher than those of the CNN. In addition, the experimental data results show that if the ViT model has more training data input, its performance seems to have a large room for improvement. Conversely, the CNN model shows no indication that increasing training data will affect its performance. Therefore, if there are more dermoscopy images with corresponding label data to make the dataset larger and balanced, the transformer may completely replace CNN on this task.