Noise annoyance due to railway traffic is a growing issue in today's society. This annoyance may be predicted using noise-exposure relationships with mean energy-based index. However, there is room for improvement of models as other acoustical and non-acoustical factors also influence noise annoyance responses. In this paper, it is proposed to highlight annoying auditory sensations evoked by the railway noise and determine acoustical and psychoacoustical indices combined in a predictive model. A laboratory experiment involving railway pass-by noise in urban areas was carried out. Annoyance ratings, noise sensitivity ratings, and free verbalization data were gathered. The analysis underlined annoying auditory sensations caused by railway pass-by noises. Two indices were proposed to account for irregular amplitude fluctuation and noise event duration-related sensations. A multilevel regression analysis was conducted, leading to two annoyance models considering noise indices and noise sensitivity. These models were finally compared to a similarly obtained multilevel regression model related to tramway noise annoyance. The comparison was carried out as a cross-validation considering the models and the respective datasets collected in laboratory conditions for model construction. Results showed that the railway noise annoyance model led to a good prediction of tramway noise annoyance and vice versa.