Background: The preoperative diagnosis of phyllodes tumors (PTs) of the breast is critical to appropriate surgical treatment. However, reliable differentiation between PT and fibroadenoma (FA) remains difficult in daily clinical practice. The purpose of this study was to investigate the utility of breast MRI texture analysis for differentiating PTs from FAs.Materials and Methods: Forty-two PTs and 42 FAs were enrolled in this retrospective study. Clinical and conventional MRI features (CCMF) and MRI texture analysis were used to distinguish between PT and FA. Texture features were extracted from the axial short TI inversion recovery T2-weighted (T2W-STIR), T1-weighted pre-contrast, and two contrast-enhanced series (first contrast and third contrast). The Mann–Whitney U test was used to select statistically significant features of texture analysis and CCMF. Using a linear discriminant analysis, the most discriminative features were determined from statistically significant features. The K-nearest neighbor classifier and ROC curve were applied to evaluate the diagnostic performance.Results: With a higher classification accuracy (89.3%) and an AUC of 0.89, the texture features on T2W-STIR outperformed the texture features on other MRI sequences and CCMF. The AUC of the combination of CCMF with texture features on T2W-STIR was significantly higher than that of CCMF or texture features on T2W-STIR alone (p < 0.05). Based on the result of the classification accuracy (95.2%) and AUC (0.95), the diagnostic performance of the combination strategy performed better than texture features on T2W-STIR or CCMF separately.Conclusions: Texture features on T2W-STIR showed better diagnostic performance compared to CCMF for the distinction between PTs and FAs. After further validation of multi-institutional large datasets, MRI-based texture features may become a potential biomarker and be a useful medical decision tool in clinical trials having patients with breast fibroepithelial neoplasms.