Herein, a colorimetric–fluorescent hybrid bifunctional nanobead with Janus structure (J-cf-HBN) was synthesized via one-pot microemulsification. Oleylamine-coated AuNPs and aggregation-induced emission luminogens (AIEgens) were suggested as building blocks to obtain high-performance colorimetric–fluorescent signals. The as-prepared J-cf-HBNs were used as a signal amplification probe to construct an immunochromatographic assay (J-cf-HBNs-ICA) platform for the ultrasensitive detection of staphylococcal enterotoxin B (SEB) in milk samples. Owing to the rational spatial distribution of AuNPs and AIEgens, the J-cf-HBNs present a highly retained photoluminescence and enhanced colorimetric signals. Combined with a pair of highly affinitive anti-SEB antibodies, the J-cf-HBN-ICA platform enabled the fast naked-eye visualization and fluorescent quantitative detection of SEB in various milk matrices. Given the advantages of the dual-mode high-performance J-cf-HBNs, the proposed strip achieved a high sensitivity for SEB qualitative determination with a visual limit of detection (LOD) of 1.56 ng mL−1 and exhibited ultrasensitivity for SEB quantitative detection with a LOD of 0.09 ng mL−1, which is 139-fold lower than that of ELISA using same antibodies. In conclusion, this work provides new insights into the construction of multimode immunochromatographic methods for food safety detection in the field.