In recent years, vibration-based structural damage identification has made significant progress by exploiting data-driven deep learning techniques, which can efficiently extract damage-sensitive features from a large amount of data. However, in some practical engineering applications, large volumes of measurement data are not readily available. This paper proposes a novel physics-guided residual neural network (PhyResNet) framework to improve the robustness and accuracy of structural damage identification under data-scarce conditions. In contrast to the state-of-the-art purely data-driven ResNet, the proposed method embedded available physics knowledge (e.g., governing equations of dynamics) of structures into the feature learning process via a novel physics-based loss function. The input-output relationship of the network is constrained to retain its physical meaning implicitly while the demand for large amounts of labeled training data is reduced. Notably, even with only 5 % of the dataset used for training, PhyResNet achieves a 13.1 % improvement in R-Value. The performance of the proposed approach is evaluated through both numerical and experimental verifications. Results demonstrate that damage localization and quantification are achieved with high accuracies and good robustness.