Concrete structures frequently manifest diverse defects throughout their manufacturing and usage processes due to factors such as design, construction, environmental conditions and distress mechanisms. In this paper, a multilevel convolutional neural network (CNN) combined with array ultrasonic testing (AUT) is proposed for identifying the locations of hole defects in concrete structures. By refining the detection area layer by layer, AUT is used to collect ultrasonic signals containing hole defect information, and the original echo signal is input to CNN for the classification of hole locations. The advantage of the proposed method is that the corresponding defect location information can be obtained directly from the input ultrasonic signal without manual discrimination. It effectively addresses the issue of traditional methods being insufficiently accurate when dealing with complex structures or hidden defects. The analysis process is as follows. First, COMSOL-Multiphysics finite element software is utilized to simulate the AUT detection process and generate a large amount of ultrasonic echo data. Next, the extracted signal data are trained and learned using the proposed multilevel CNN approach to achieve progressive localization of internal structural defects. Afterwards, a comparative analysis is conducted between the proposed multilevel CNN method and traditional CNN approaches. The results show that the defect localization accuracy of the proposed multilevel CNN approach improved from 85.38% to 95.27% compared to traditional CNN methods. Furthermore, the computation time required for this process is reduced, indicating that the method not only achieves higher recognition precision but also operates with greater efficiency. Finally, a simple experimental verification is conducted; the results show that this method has strong robustness in recognizing noisy ultrasonic signals, provides effective solutions, and can be used as a reference for future defect detection.