The rapid sampling and efficient collection of target molecules from a real-world surface is fairly crucial for surface-enhanced Raman scattering (SERS) to detect trace pesticide residues in the environment and in agriculture fields. In this work, a versatile approach was exploited to fabricate a flexible SERS substrate for highly sensitive detection of carbaryl pesticides, using in-situ grown silver nanoparticles (AgNPs)on non-woven (NW) fabric surfaces based on mussel-inspired polydopamine (PDA) molecules. The obtained NW@PDA@AgNPs fabrics showed extremely sensitive and reproducible SERS signals toward crystal violet (CV) molecules, and the detection limit was as low as 1.0 × 10−12 M. More importantly, these NW@PDA@AgNPs fabrics could be directly utilized as flexible SERS substrates for the rapid extraction and detection of trace carbaryl pesticides from various fruit surfaces through a simple swabbing approach. It was identified that the detection limits of carbaryl residues from apple, orange, and banana surfaces were approximately decreased to 4.02 × 10−12, 6.04 × 10−12, and 5.03 × 10−12 g, respectively, demonstrating high sensitivity and superior reliability. These flexible substrates could not only drastically increase the collection efficiency from multifarious irregular-shaped matrices, but also greatly enhance analytical sensitivity and reliability for carbaryl pesticides. The fabricated flexible and multifunctional SERS substrates would have great potential to trace pesticide residue detection in the environment and bioscience fields.