Transportation engineers face challenging safety investment decisions, particularly for highway-railway grade crossings (HRGCs), where rare collision occurrences and incomplete historical records complicate the assessment of countermeasure cost-effectiveness. This study introduces an empirical Bayes (EB) observational before–after approach to address these challenges, specifically examining the impact of a widely used safety countermeasure: flashing lights, bells, and gates (FLBG). The research covers a total 171 FLBG crossings with pre-implementation conditions varying between standard reflective crossing signs and flashing lights and bells, utilizing collision data from 2000 to 2012 for analysis. This study is the first to substantiate quantitatively the effectiveness of FLBG treatment in improvising the safety of HRGCs and show that the effect is dependent of crossing characteristics, such as track angle, pavement conditions, and train speed limits. These findings can provide valuable support to decision-makers, enabling them to make more effective decisions for grade crossing safety improvement projects.