Mammography is an important imaging technique for the detection of early breast cancer. Doctors classify mammograms as Breast Imaging Reporting and Data Systems (BI-RADS). This study aims to provide an intelligent BI-RADS grading prediction method, which can help radiologists and clinicians to distinguish the most challenging 4A, 4B, and 4C cases in mammography. Firstly, the breast region, the lesion region, and the corresponding region in the contralateral breast were extracted. Four categories of features were extracted from the original images and the images after the wavelet transform. Secondly, an optimized sequential forward floating selection (SFFS) was used for feature selection. Finally, a two-layer classifier integration was employed for fine grading prediction. 45 cases from the hospital and 500 cases from Digital Database for Screening Mammography (DDSM) database were used for evaluation. The classification performance of the support vector machine (SVM), Bayes, and random forest is very close on the 45 testing set, with the area under the receiver operating characteristic curve (AUC) of 0.978, 0.967, and 0.968. On the DDSM set, the AUC achieves 0.931, 0.938, and 0.874. Using the mean probability prediction, the AUC on the two datasets reaches 0.998 and 0.916. However, they are all significantly higher than the doctors' diagnosis, with the AUC of 0.807 and 0.725. A BI-RADS fine grading (2, 3, 4A, 4B, 4C, 5) prediction model was proposed. Through the evaluation from different datasets, the performance is proved higher than that of the doctors, which may provide great help for clinical BI-RADS classification diagnosis. Therefore, our method can produce more effective and reliable results.