Casein is rich in immunomodulatory peptides. In this study, the release of casein-derived immunomodulatory peptides by Kluyveromyces marxianus JY-1 was investigated for the first time, and an immunosuppressive mouse model was used to evaluate the immunomodulatory activity in the casein hydrolysate. The results showed that the cellular and humoral immunity of immunosuppressed mice could be significantly enhanced by casein hydrolysate. Peptide HPHPHLSF with high immunomodulatory activity from casein hydrolysate was screened using the virtual screening technique. HPHPHLSF possessed strong immunomodulatory activity and significantly upregulated the expression of IL-6, IL-1β, and TNF-α. Next, the interaction of HPHPHLSF with TLR2/4 on the cell surface of RAW264.7 cells was further elucidated by molecular docking and combined analysis of double-stranded small interfering RNA and receptor inhibitors. Further, the results of the highest occupied molecular orbital energy distribution elucidated that the histidine active site C48═O49 played an important role in the immunomodulatory activity of HPHPHLSF. This study confirmed that casein hydrolyzed by K. marxianus JY-1 was a natural immunomodulator, while the structure-activity relationship analysis provided new theoretical and technical support for the targeted preparation and screening of casein-derived immunomodulatory peptides.