The Shanhaiguan Great Wall is a section of the Great Wall of the Ming Dynasty, which is a UNESCO World Heritage Site. Both sides of its basic structure are composed of rammed earth and gray bricks. The surface gray bricks sustain damage from environmental factors, resulting in a decline in their structural quality and even a threat to their safety. Traditional surface damage detection methods rely primarily on manual identification or manual identification following unmanned aerial vehicle (UAV) aerial photography, which is labor-intensive. This paper applies the YOLOv4 machine learning model to the gray surface bricks of the Plain Great Wall of Shanhaiguan as an illustration. By slicing and labeling the photos, creating a training set, and then training the model, the proposed approach automatically detects four types of damage (chalking, plants, ubiquinol, and cracking) on the surface of the Great Wall. This eliminates the need to expend costly human resources for manual identification following aerial photography, thereby accelerating the work. Through research, it is found that 1) compared with manual detection, this method can quickly and efficiently monitor a large number of wall samples in a short period of time and improve the efficiency of brick wall detection in ancient buildings. 2) Compared with previous approaches, the accuracy of the current method is improved. The identifiable types are increased to include chalking and ubiquinol, and the accuracy rate increases by 0.17% (from 85.70% before to 85.87% now). 3) This method can quickly identify the damaged parts of the wall without damaging the appearance of the historical building structure, enabling timely repair measures.