We propose an anti-scratch flexible surface-enhanced Raman scattering substrate with arrayed nanocavity microstructures fabricated by colloidal lithography. The nanocavity microstructure of the substrate can well protect the inner gold nanoparticles during wipe sampling. The prepared flexible substrate was able to detect 4-aminothiophenol (4-ATP) with a concentration down to 1 fM. Furthermore, the substrate was used to detect 6-BA residues on the surface of apples and bean sprouts through wipe sampling, which shows great potential in the field of rapid on-site detection, especially in the detection of pesticide residues on the surface of fruits and vegetables.