In this study, we aimed to explore if the combination of tumor infiltrating lymphocytes (TILs) and change in tumor load on dynamic contrast-enhanced magnetic resonance imaging leads to better assessment of response to neoadjuvant chemotherapy (NAC) in patients with breast cancer, compared to either alone. In 190 NAC treated patients, MRI scans were performed before and at the end of treatment. The percentage of stromal TILs (%TILs) was assessed in pre-NAC biopsies according to established criteria. Prediction models were developed with linear regression by least absolute shrinkage and selection operator and cross validation (CV), with residual cancer burden as the dependent variable. Discrimination for pathological complete response (pCR) was evaluated using area under the receiver operating characteristic curves (AUC). We used Cox regression analysis for exploring the association between %TILs and recurrence-free survival (RFS). Fifty-one patients reached pCR. In all patients, the %TILs model and change in MRI tumor load model had an estimated CV AUC of 0.69 (95% confidence interval (CI) 0.53-0.78) and 0.69 (95% CI 0.61-0.79), respectively, whereas a model combining the variables resulted in an estimated CV AUC of 0.75 (95% CI 0.66-0.83). In the group with tumors that were ER positive and HER2 negative (ER+/HER2-) and in the group with tumors that were either triple negative or HER2 positive (TN&HER2+) separately, the combined model reached an estimated CV AUC of 0.72 (95% CI 0.60-0.88) and 0.70(95% CI 0.59-0.82), respectively. A significant association was observed between pre-treatment %TILS and RFS (hazard ratio (HR) 0.72 (95% CI 0.53-0.98), for every standard deviation increase in %TILS, p = 0.038). The combination of TILs and MRI is informative of response to NAC in patients with both ER+/HER2- and TN&HER2+ tumors.