Computer-aided diagnostic systems developed with Artificial Intelligence (AI) are a major advancement in healthcare analytics, assisting radiologists with a second opinion on cancer diagnosis. In this study, we explore two modalities of breast images, ultrasound, and histology, for their classification into cancerous and non-cancerous categories. Traditionally Convolution Neural Networks (CNN) have done a commendable job of extracting features using convolution kernels from images with great accuracy. Also, the Bidirectional Encoder Representations from Transformers (BERT) has been widely used in Natural Language Processing (NLP) for feature encoding and downstream tasks like segmentation and classification. We extract the power of Vision Transformers (ViT) and implement transfer learning using BERT pre-training of Image Transformers (BEiT) as a feature encoding technique. We use the encoded features for classification with Recurrent Neural Network – Long short Term Memory (RNN-LSTM). The classification is performed on two modalities of breast image datasets: BUSI1311 and breast histopathological images. Both modalities yielded competitive accuracies. The BUSI1311 dataset produced 99 percent accuracy compared to 91 percent accuracy for breast histology images.