To explore the potential factors related to the pathological grade of breast phyllodes tumors (PTs) and to establish a nomogram to improve their differentiation ability. Patients with PTs diagnosed by post-operative pathology who underwent pretreatment magnetic resonance imaging (MRI) from January 2015 to June 2020 were retrospectively reviewed. Traditional clinical features and MRI features evaluated according to the fifth BI-RADS were analyzed by statistical methods and introduced to a stepwise multivariate logistic regression analysis to develop a prediction model. Then, a nomogram was developed to graphically predict the probability of non-benign (borderline/malignant) PTs. Finally, 61 benign, 73 borderline and 48 malignant PTs were identified in 182 patients. Family history of tumor, diameter, lobulation, cystic component, signal on fat saturated T2 weighted imaging (FS T2WI), BI-RADS category and time-signal intensity curve (TIC) patterns were found to be significantly different between benign and non-benign PTs. The nomogram was finally developed based on five risk factors: family history of tumor, lobulation, cystic component, signal on FS T2WI and internal enhancement. The AUC of the nomogram was 0.795 (95% CI: 0.639, 0.835). Family history of tumor, lobulation, cystic components, signals on FS T2WI and internal enhancement are independent predictors of non-benign PTs. The prediction nomogram developed based on these features can be used as a supplemental tool to pre-operatively differentiate PTs grades. More sample size and characteristics were used to explore the factors related to the pathological grade of PTs and establish a predictive nomogram for the first time.