COVID-Attention: Efficient COVID19 Detection Using Pre-trained Deep Models Based on Vision Transformers and X-ray Images

Abstract

COVID19 is becoming more and more threatening to human life since the appearance of many variants like Alpha, Omicron and Delta. These variants have dozens of mutations that can make their diagnosis more challenging. Despite the recent success of Convolutional Neural Networks (CNN) to detect COVID19 automatically using transfer learning techniques, they are not the most robust to accomplish this task since some translation, scale and hyperparameter variations can affect the accuracy. Newly, Vision Transformers (ViT) are becoming increasingly popular to handle similar tasks and to deal with the aforementioned variations. In this paper, we propose an enhanced ViT architecture for COVID19 detection referred to as COVID-Attention. The proposed ViT-based models are robust due to their capability to capture long-range dependencies within images thanks to the attention mechanism, which is the core of the transformer block. We compare the efficiency of our proposed method to top-performing CNN baselines using two different transfer learning modes. We further show in our experiments that adding a convolution block to the top of the ViT model (i.e. as an initial block) can avoid the collapse issue and enhance the ViT performance using the recent ViTC model. We finally show that ViT-based models give more explainable visualization compared to CNN models using the Grad-CAM technique in order to highlight the attention map that affects the classification decision. Our experiments have been conducted on two recent databases of X-ray images and show high performance compared to the state-of-the-art methods in three-class classifications.