Objective To investigate the clinicopathological features of 166 cases of invasive ductal carcinoma (IDC) of the breast and to analyze the effect of the location of the primary tumor on the prognosis of modified radical mastectomy. Materials and Methods The clinical data of 166 patients with IDC who underwent modified radical mastectomy in our hospital from May 2015 to May 2017 were retrospectively analyzed. The clinicopathological features of IDC patients were recorded. Univariate analysis and the multivariate logistic regression model were used to analyze the relationship between the location of the primary tumor and the prognosis of IDC patients after modified radical surgery. The effect of primary tumor location on the prognosis of modified radical resection was used with Survival curve analysis. Results Among the patients in the central region, 13.33% had tumors >5 cm in diameter, which was higher than those in the other four groups. Among the patients in the upper inner quadrant, 59.38% received hormone therapy after operation, which was higher than those in the other four groups (P < 0.05). There were no significant differences in age, menopause, histological grading, molecular typing, lymph node metastasis, vascular invasion, radiation therapy, and chemotherapy among different groups (P > 0.05). Univariate analysis showed that molecular typing, lymph node metastasis, vascular invasion, and location of the primary tumor were all related to the prognosis of IDC patients after modified radical surgery, and the differences were statistically significant (P < 0.05). Logistic regression analysis showed that molecular typing, lymph node metastasis, vascular invasion, and primary tumor location were all independent influencing factors for prognosis of IDC patients after modified radical surgery (P < 0.05). As of 31 May 2021, there were 11 patients with recurrence and metastasis and 20 patients with death. The median survival time in the outer upper quadrant group was 80 months, which was higher than that in the outer lower quadrant group by 72 months, the median survival time in the central region group by 71 months, the median survival time in the inner upper quadrant group by 67 months, and the median survival time in the inner lower quadrant group by 61 months. The log-rank test showed all P < 0.001. Conclusion Patients with primary tumors located in the central area have larger tumor diameters. Patients located in the central area, upper inner quadrant, and lower inner quadrant are more likely to have lymphatic metastasis, have a more serious condition, and have a shorter prognosis survival time. Unluminal type, multiple lymph node metastases, vascular invasion, and the location of the primary tumor in the inner quadrant are all independent risk factors for prognosis in patients after modified radical surgery for IDC.