Ag-27.9 at% Co-8.9 at% Zr alloy films were deposited on flexible polyimide (PI) by magnetron sputtering and annealed in argon atmosphere. Results show that many fine nanoparticles are formed on the surface of as-deposited Ag–Co–Zr alloy films. Furthermore, the average size of self-formed Ag nanoparticles gradually increases while the number of Ag nanoparticles decreases as alloy film thickness increases. In addition, it is worth noting that flake-like and polyhedral nanoparticles are formed spontaneously on the surface of annealed alloy films and are proved to be single crystal Ag nanoparticles. Moreover, the morphology of flake-like Ag nanoparticles is dependent on annealing temperature. In order to improve “hotspots” activity of the alloy films, a 12 nm pure Ag layer was deposited on the annealed alloy film surface to obtain Ag layer/Ag nanoparticles/Ag–Co–Zr alloy films SERS substrates which can detect 5 × 10−14 M R6G. This paper provides a potential candidate for preparation of SERS substrates with ultra-high sensitivity and stability.