Facile fabrication of flexible substrates containing high concentration of nanoparticles (NPs) is very promising owing to their capability of swab-based surface-enhanced Raman scattering (SERS) trace detection. However, the background signal of the substrate could compromise the trace-detection capabilities. Moreover, the presence of fluorescent molecules may result in intense fluorescence background which could overshadow the Raman peaks. Herein, we demonstrate that the surfactant-free bimetallic (Ag and Au) NPs, synthesized directly on the filter paper, are very effective in reducing the background signal in SERS-based trace detection. Simple soaking of filter paper in a mixture of HAuCl4 and AgNO3 solution, and its immediate drying and reduction produces Au-Ag alloy NPs on the filter paper. Interestingly, the substrates are very effective in quenching the background fluorescence of both filter paper and the analyte, thereby effectively detecting the SERS signature of the fluorescent molecules. In particular, Rhodamine6G (R6G) concentrations down to 10−10 M were detected under resonance excitation, both by solution drying and swabbing. The swab detection of ammonium nitrate, which is usually used in homemade explosives, was also demonstrated. Moreover, owing to the fluorescence quenching properties, the Au-Ag alloy substrates (AuAgS) were able to carry out the swab-based multiplex-detection of crystal violet (CV), Brilliant Cresyl Blue (BCB), and R6G upon resonance excitation of R6G. Finally, the substrates have shown good reproducibility, stability, and signal uniformity. This verifies the potential of AuAgS for real-world trace-detection applications.