Abstract This study presents an application of Long Short-Term Memory Autoencoder (LSTM AE) for the detection of broken rails based on laser Doppler Vibrometer (LDV) measurements. This work is part of an ongoing project aimed at developing a non-contact damage detection system using LDV measurements. The damage detection system consists of two laser Doppler vibrometers (LDV) mounted on a moving rail car to measure vibrations induced on the rail head. Field tests were carried out at the Transportation Technology Center (TTC) in Pueblo, CO, to collect the vibrational data. This study focused on the detection of broken rails. To simulate the reflected and transmitted waves induced by the broken rail, a welded joint was used. The data was collected from moving LDV measurements, in which the train was operating at three different speeds: 16km/h (10mph), 32km/h (20mph), and 48km/h (30mph). After obtaining the data, filtering and signal processing were applied to obtain the signal features in time and frequency domains. Next, correlation analysis and principal component analysis were carried out for feature selection and dimension reduction to determine the input used to train and test the LSTM AE model. In this study, the LSTM AE models were trained based on different data sets for anomaly detection. Consequently, an automatic anomaly detection approach for anomaly detection based on the LSTM AE model was evaluated. The results show that the LSTM AE model can efficiently detect the anomaly based on the selected features at three different speeds.