Background Impact of previous radiation therapy and neoadjuvant chemotherapy (NACT) on early complications in direct-to-implant (DTI) breast reconstruction has not been elucidated. This study investigated whether DTI reconstruction is viable in patients with NACT or a history of preoperative chest wall irradiation. Methods Medical records of breast cancer patients who underwent nipple-sparing or skin-sparing mastectomy with DTI breast reconstruction from March 2018 to February 2021, with at least 1 year of follow-up in a single tertiary center, were reviewed. Demographic data, intraoperative details, and postoperative complications, including full-thickness necrosis, infection, and removal, were reviewed. Risk factors suggested by previous literature, including NACT and preoperative chest wall irradiation histories, were reviewed by multivariate analysis. Results A total of 206 breast cancer patients were included, of which, 9 were bilateral, 8 patients (3.9%) had a history of prior chest wall irradiation, and 17 (8.6%) received NACT. From 215 cases, 11 cases (5.1%) required surgical intervention for full-thickness necrosis, while intravenous antibiotics or hospitalization was needed in 11 cases (5.1%), with 14 cases of failure (6.5%) reported. Using multivariable analysis, preoperative irradiation was found to significantly increase the risk of full-thickness skin necrosis (OR = 12.14, p = 0.034), and reconstruction failure (OR = 13.14, p = 0.005). NACT was not a significant risk factor in any of the above complications. Conclusion DTI breast reconstruction is a viable option for patients who have received NACT, although reconstructive options should be carefully explored for patients with a history of breast irradiation.