Automated and information-based inspection methods are urgently needed for the tunnel’s daily maintenance as the number of tunnels continues to increase. A remotely non-contact detection method is proposed using a laser Doppler vibrometer based on ambient vibration. It introduces a dimensionless quantitative index called Fundamental Frequency Deterioration Degree (FFDD) based on the fundamental frequency. The FFDD is developed from damage mechanics and vibration mechanics, enabling a more accurate assessment. To validate this method, laboratory ambient vibration detection tests were conducted on cubic concrete samples considering strength, cavity, and cracks under compression. The PDV-100 laser Doppler vibrometer was used for ambient vibration collection. Results demonstrate a strong correlation between concrete defects and the fundamental frequency. The relationship between FFDD and concrete defects validates its accuracy. The method was applied in situ to detect and assess grouting defects in a newly constructed metro tunnel, further validating its use in real-world scenarios.