Virulent Listeria monocytogenes is responsible for listeriosis, a highly fatal foodborne illness that impairs human health. One of the major causes of its pathogenicity is the carriage of virulence factor, listeriolysin O, encoded by hylA gene. Herein, the specificity and conservativity of hylA gene sequences were analyzed, with most conservative region selected. Then, a crossing priming amplification (CPA) assay targeting hylA conservative region was established under an isothermal condition (63 °C) to identify virulent L. monocytogenes within 60 min. Forty-four reference strains including two standard L. monocytogenes strains and forty-two strains of other species were selected to assess sensitivity and specificity. The CPA assay showed strong specificity and sensitivity (genomic DNA at 5.66 pg/μL). The practicality of the CPA assay was further confirmed in four artificially contaminated rice-flour products with limit of detection at 104 CFU/mL. In addition, a PMA-CPA assay was developed to direct detect L. monocytogenes viable cells with rice-flour products as a model system. In conclusion, the developed CPA method has advantages of high sensitivity, specificity and quick measurement in the detection of virulent L. monocytogenes, showing considerably feasible promise for future food safety.