Holographic microwave imaging (HMI) has been proposed for early breast cancer diagnosis. Automatically classifying benign and malignant tumors in microwave images is challenging. Convolutional neural networks (CNN) have demonstrated excellent image classification and tumor detection performance. This study investigates the feasibility of using the CNN architecture to identify and classify HMI images. A modified AlexNet with transfer learning was investigated to automatically identify, classify, and quantify four and five different HMI breast images. Various pre-trained networks, including ResNet18, GoogLeNet, ResNet101, VGG19, ResNet50, DenseNet201, SqueezeNet, Inception v3, AlexNet, and Inception-ResNet-v2, were investigated to evaluate the proposed network. The proposed network achieved high classification accuracy using small training datasets (966 images) and fast training times.