We aimed to develop an ultrasound-based radiomics model to distinguish between sclerosing adenosis (SA) and invasive ductal carcinoma (IDC) to avoid misdiagnosis and unnecessary biopsies. From January 2020 to March 2022, 345 cases of SA or IDC that were pathologically confirmed were included in the study. All participants underwent pre-surgical ultrasound (US), from which clinical information and ultrasound images were collected. The patients from the study population were randomly divided into a training cohort (n = 208) and a validation cohort (n = 137). The US images were imported into MaZda software (Version 4.2.6.0) to delineate the region of interest (ROI) and extract features. Intragroup correlation coefficient (ICC) was used to evaluate the consistency of the extracted features. The least absolute shrinkage and selection operator (LASSO) logistic regression and cross-validation were performed to obtain the radiomics score of the features. Based on univariate and multivariate logistic regression analyses, a model was developed. 56 cases from April 2022 to December 2022 were included for independent validation of the model. The diagnostic performance of the model and the radiomics scores were evaluated by performing the receiver operating characteristic (ROC) analysis. The calibration curve and decision curve analysis (DCA) were used for calibration and evaluation. Leave-One-Out Cross-Validation (LOOCV) was used for the stability of the model. Three predictors were selected to develop the model, including radiomics score, palpable mass and BI-RADS. In the training cohort, validation cohort and independent validation cohort, AUC of the model and radiomics score were 0.978 and 0.907, 0.946 and 0.886, 0.951 and 0.779, respectively. The model showed a statistically significant difference compared with the radiomics score (p<0.05). The Kappa value of the model was 0.79 based on LOOCV. The Brier score, calibration curve, and DCA showed the model had a good calibration and clinical usefulness. The model based on radiomics, ultrasonic features, and clinical manifestations can be used to distinguish SA from IDC, which showed good stability and diagnostic performance. The model can be considered a potential candidate diagnostic tool for breast lesions and can contribute to effective clinical diagnosis.