Abstract Defect recognition is the key to realizing automatic visual inspection and quality control in intelligent manufacturing. Deep learning (DL) has been widely applied to locate damaged areas in images, characterize impurities of materials and further analyze the quality of products. However, automatic defect detection by DL in practical industrial applications is still a challenge due to the lack of sufficient datasets and appropriate recognition methods. Here, a novel Attention Pyramid Dilated Region-based Convolutional Neural Network (ADRCNN) is proposed to realize the multi-scale defect recognition and pixel-level instance segmentation of metallographic images. We also collected a dataset containing 900 images of commercial A356 aluminum alloy with different casting defects. The ADRCNN model is evaluated on our dataset with an average detection accuracy of 87.2% and 93.3% on validation and test sets respectively. To address the overfitting problem, a pretraining fine-tuning strategy is implemented by using the pretraining weights from large-scale datasets and temporarily freezing the weights of the backbone network during the training process. The experimental results show that the proposed model can achieve satisfactory defect segmentation in metallographic images, promoting the development of intelligent manufacturing in alloy industries.