In recent years, nondestructive testing for civil engineering structures has become increasingly important. Ultrasonic testing is one of nondestructive inspection methods for civil structures. However, the inspection of civil engineering structures takes much time because of the extensive scope of the inspection. Moreover, in the field of nondestructive testing, there are also concerns about a future shortage of inspectors, so that an innovative effective nondestructive method needs to be developed. This study proposes an automatic defect detection approach using pre-trained convolutional neural network for laser ultrasonic visualization testing. The effectiveness of the proposed method is confirmed by applying it to a concrete structure with a surface defect. Grad-CAM demonstrates that the created CNN model in this study accurately predicts the position of a surface defect of concrete specimens.