Accurate damage identification is of great significance to maintain timely and prevent structural failure. To accurately and quickly identify the structural damage, a novel two-stage approach based on convolutional neural networks (CNN) and an improved hunter–prey optimization algorithm (IHPO) is proposed. In the first stage, the cross-correlation-based damage localization index (CCBLI) is formulated using acceleration and is input into the CNN to locate structural damage. In the second stage, the IHPO algorithm is applied to optimize the objective function, and then the damage severity is quantified. A numerical model of the American Society of Civil Engineers (ASCE) benchmark frame structure and a test structure of a three-storey frame are adopted to verify the effectiveness of the proposed method. The results demonstrate that the proposed approach is effective in locating and quantifying structural damage precisely regardless of noise perturbations. In addition, the reliability of the proposed approach is evaluated using a comparison between it and approaches based on CNN or the IHPO algorithm alone. The comparison results indicate that in single and multiple damage events, the proposed two-stage damage identification approach outperforms the other two approaches on the accuracy, and the average consumption time is 20% less than the method using the IHPO algorithm alone. Therefore, this paper provides a guideline for the study of high-accuracy and quick damage identification using both data-based and model-based hybrid methods.