Introduction: Compromised neonatal intensive care unit neonates are at risk of acquiring late-onset infections (late-onset sepsis [LOS]). Neonates born with congenital anomalies (CAs) could have an additional LOS risk. Methods: Utilising the population-based Australian and New Zealand Neonatal Network data from 2007 to 2017, bacterial LOS rates were determined in very preterm (VPT, <32 week), moderately preterm (MPT, 32–36 weeks), and term (FT, 37–41 weeks) neonates with or without CA. Stratified by major surgery, the association between CA and bacterial LOS was evaluated. Results: Of 102,808 neonates, 37.7%, 32.8%, and 29.6% were born VPT, MPT, and FT, respectively. Among these, 3.4% VPT, 7.5% MPT, and 16.2% FT neonates had CA. VPT neonates had the highest LOS rate (11.1%), compared to MPT (1.8%) and FT (1.8%) neonates. LOS rates were higher in CA neonates than those without (8.2% versus 5.1% adjusted relative risk [aRR] 1.67, 95% confidence interval [CI]: 1.45–1.92). Neonates with surgery had a higher LOS rate (14.2%) than neonates without surgery (4.4%, p < 0.001). Among the neonates without surgery, CA neonates had consistently higher LOS rates than those without CA (VPT 14.3% vs. 9.6% [aRR 1.32, 95% CI: 1.11–1.57]; MPT 4% vs. 0.9% [aRR 4.45, 95% CI: 3.23–6.14]; and FT 2% vs. 0.7% [aRR 2.87, 95% CI: 1.97–4.18]). For the neonates with surgery, CAs were not associated with additional LOS risks. Conclusion: Overall, we reported higher rates of LOS in neonates with CA compared to those without CA. Regardless of gestation, CA was associated with an increased LOS risk among non-surgical neonates. Optimisation of infection prevention strategies for CA neonates should be explored. Future studies are needed to evaluate if the infection risk is caused by CA or associated complications.