Non-specific binding in fluorescence resonance energy transfer (FRET) remains a challenge in foodborne pathogen detection, resulting in interference of high background signals. Herein, we innovatively reported a dual-mode FRET sensor based on a “noise purifier” for the ultrasensitive quantification of Escherichia coli O157:H7 in food. An efficient FRET system was constructed with polymyxin B-modified nitrogen-sulfur co-doped graphene quantum dots (N, S-GQDs@PMB) as donors and aptamer-modified yellow carbon dots (Y-CDs@Apt) as acceptors. Magnetic multi-walled carbon nanotubes (Fe@MWCNTs) were employed as a “noise purifier” to reduce the interference of the fluorescence background. Under the background purification mode, the sensitivity of the dual-mode signals of the FRET sensor has increased by an order of magnitude. Additionally, smartphone-assisted colorimetric analysis enabled point-of-care detection of E. coli O157:H7 in real samples. The developed sensing platform based on a “noise purifier” provides a promising method for ultrasensitive on-site testing of trace pathogenic bacteria in various foodstuffs.