Mutation study for high-risk breast and ovarian cancer (HBOC) has been extensively studied in patients of different ethnicities. Here we compared the germline mutation rate and mutation spectrum of patients (n = 4341) with benign breast diseases or breast cancers, with and without other risk factors. Three cohorts of Chinese patients were recruited. The first cohort, high-risk cohort (HR, n = 3935) included high-risk breast cancer patients fulfilling high-risk HBOC criteria and who are recruited at our genetics clinic. The second cohort, unselected cancer cohort (CC, n = 307) was from general recruitment of patients with breast cancer at breast surgery clinics. The third cohort, benign breast lesion cohort (NC, n = 99) comprised 99 patients with benign breast diseases such as fibroadenoma, fibroadenomatoid hyperplasia, and intraductal papilloma. Thirty HBOC related genes were sequenced on the above-mentioned patient cohorts. The germline mutation rates of HR, CC, and NC cohort were 11.9%, 6.5%, and 8.1%, respectively. In the CC cohort, 29.3% (90/307) of patients fulfilled the National Comprehensive Cancer Network (NCCN) high-risk genetic test criteria 2022 v.2. The mutation rate for this group of patients was 11.1%, similar to that of the HR cohort, while the mutation rate for those not fulfilling testing criteria was 4.6%, like that of the NC cohort. High penetrance genes (BRCA1/2, CDH1, PALB2, PTEN, and TP53) mutations were only found in the HR (10.6%) and CC (3.3%) cohorts but were not found in the NC cohort. ATM, BRIP1, RAD51C, and RAD51D mutations were identified in all cohorts. RAD51C and RAD51D mutations showed conflicting penetrance. An unexpectedly high mutation rate of total 2% was found in the NC cohort but it was only 0.3% and 0.5% in the HR cohort and CC cohort, respectively. Our results show a clinical need to enhance genetic testing of unselected breast cancer patients to identify the high-risk patients.