In the structural health monitoring of vibration systems, varying excitation always affects the accuracy of damage identification. The proposed symbolic three-order square matrix damage detection method with the matrix norm as a damage indicator can solve the difficult problem of damage identification under ambient excitation. The new sampling pattern extracts data from signals in the time domain at specific intervals based on the structural properties with the help of the autocorrelation coefficient. Then, the data extracted are converted into symbols and arranged into a three-order square matrix, and the Frobenius norm of the matrix is used for structural damage identification as a reliable damage indicator. In this process, the transmissibility function is employed to eliminate the effects of varying excitation. First, the method was verified by a cracked simply supported beam-a simulated Abaqus model. Then, a wooden truss bridge in the laboratory and an actual engineering scenario under ambient excitation together demonstrated the effectiveness and accuracy of the damage identification method and proved the proposed method to be robust to different types of damage under ambient excitation. Compared with other related methods, this method is more intuitive and efficient.