Structural defect identification is a vital aspect of structural health monitoring used to assess the safety of engineering structures. However, quantitatively determining the dimensions of structural defects is often difficult. Therefore, this study presents an innovative data-driven algorithm that combines the scaled boundary finite element method (SBFEM) and a deep learning framework based on a dilated causal convolutional neural network (CNN) to identify crack-like defects in large-scale structures. The SBFEM is used to simulate different crack-like defects. Mesh generation is significantly simplified by a simple procedure that requires only changing the scale centre at the crack tip and the positions of the nodes at the crack opening. This minimises remeshing and enables simple generation of sufficient data to train the neural network. In addition, an absorbing boundary model based on Rayleigh damping is used to avoid computing the entire model when simulating wave propagation in massive structures. To ensure that sequential data remain ordered and to obtain a large receptive field without increasing the complexity of the neural network, a dilated causal CNN is employed in the deep learning framework. Therefore, more historical information is captured, and the complex mapping relationship between the echo signal and the crack information is efficiently learnt. The proposed model can accurately identify the number, location, and depth of cracks in massive structures. Moreover, it is robust to noise, which is demonstrated via numerical examples. Therefore, the proposed algorithm provides valuable insight into the detection and diagnosis of structural defects, which can ultimately improve the safety of engineering structures.