In this work, the origin of long-term stability in the silver nanorods (Ag NRs) feasible for surface-enhanced Raman spectroscopy (SERS) substrate was investigated. The Ag NRs films were deposited on silicon wafer substrates by magnetron sputtering with the glancing-angle deposition (GLAD) technique. It was found that the background contamination in the SERS spectra was suppressed by storing the as-deposited Ag NRs samples under the high vacuum condition after the deposition. Furthermore, the SERS response remained stable in an air ambient usage despite a slight decrease in the Raman signal. The stored Ag NRs under high vacuum exhibited long-term SERS stability over three months. X-ray photoemission spectroscopy revealed that the carbon was accumulated on the Ag NRs film surface after the high vacuum storage, and it prolonged the SERS performance.