Listeria monocytogenes is an important foodborne pathogenic bacterium that is explicitly threatening public health and food safety. Rapid, simple, and sensitive detection methods for this pathogen are of urgent need for the increasing on-site testing demands. Application of the isothermal recombinase polymerase amplification (RPA) and the lateral flow strip (LFS) in the detection is promising for fast speed, high sensitivity, and little dependency on equipment and trained personnel. However, the simplicity comes with an intrinsic and non-negligible risk, the false-positive signals from primer–dimers. In this study, an improved RPA–LFS system was established for detection of L. monocytogenes that eliminated false-positive signals from primer–dimers. Primer candidates were carefully selected from the entire L. monocytogenes genome sequence and rigorously screened for specific amplifications in PCR and RPA reactions. For the optimal primer pairs, probes that matched the targeted fragment sequences, although had the smallest chance to form cross-dimers with the primers, were designed and screened. The intelligent use of the probe successfully linked the positive signal to the actual amplification product. This RPA–LFS system was highly specific to L. monocytogenes and was able to detect as low as 1 colony-forming unit of the bacterium per reaction (50 μl) without DNA purification, or 100 fg of the genomic DNA/50 μl. The amplification could be conducted under the temperature between 37 and 42°C, and the whole detection finished within 25 min. Test of artificially contaminated milk gave 100% accuracy of detection without purification of the samples. Various food samples spiked with 10 colony-forming unit of L. monocytogenes per 25 g or 25 ml were successfully detected after an enrichment time period of 6 h. The RPA–LFS system established in this study is a rapid, simple, and specific detection method for L. monocytogenes that has eliminated false-positive results from primer–dimers. In addition, this study has set a good example of eliminating the false-positive risk from primer–dimers in isothermal amplification-based detection methods, which is applicable to the development of detection technologies for other pathogens.