Rubber bearing condition evaluation is crucial for bridge inspection, and the current practice heavily relies on human-vision inspection. Convolutional neural networks (CNNs) have shown great potential for structured damage recognition tasks in recent years; however, this method usually requires a large training data set, which is difficult to collect in practice for rubber bearings. Therefore, methods to improve the performance of CNN for condition classification for elastomeric bearings are necessary. In this paper, a geometric attention regularization (GAR) method is proposed to enhance the performance of CNN for the condition evaluation of rubber bearings. Firstly, the data set of bearings contains different damages that are collected and labeled where the location of the rubber bearing is presented as a bounding box. Then, the location information is utilized to enhance the loss function of CNN in two aspects. On one hand, the bearing location worked as an attention mechanism to indicate the important part of the input image. Besides, it worked as a regularization method to mitigate the effect of overfitting. Experiments using two CNN architectures, including VGG-11 and ResNet-18 trained with transfer learning techniques, are used to evaluate the efficacy of the proposed method. The results show the proposed method is effective to enhance the performance of the CNN model.