Staphylococcal food poisoning is a significant foodborne illness caused by staphylococcal enterotoxins (SEs). Immunoassays have become the primary method for rapidly detecting harmful bacteria and toxins because of their excellent sensitivity and specificity. However, these assays have limitations in that they cannot differentiate between types of SEs and do not provide rapid, on-site, quantitative testing. In this study, a time-resolved fluorescence immunochromatography assay (TRFICA) was developed specifically for detecting staphylococcal enterotoxin E (SEE), which is commonly found in dairy products. Compared with a double antibody sandwich enzyme-linked immunosorbent assay, which had a detection limit of 0.028 ng/mL, TRFICA demonstrated comparable sensitivity, enabling SEE quantification with a detection limit as low as 0.081 ng/mL in infant formula. Validation by spiking infant formula samples confirmed no cross-reactivity with analogs (recoveries ranged from 93.17% to 128.77%). Furthermore, with an 8-min reaction time and interpretation delivered by a portable TRFICA strip reader, our method demonstrates potential for use in mobile and on-site detection. This study describes a rapid, easy, and reliable method for detecting trace levels of SEE in infant formula, which could serve as an early screening tool toward preventing food poisoning in infants and children.