Impact toughness degradation and microstructure evolution of a carbon and nitrogen-controlled 316 stainless steel during long-term aging at 650 °C for up to 8800 h were investigated. The degradation of impact toughness during aging can be divided into three stages: a rapid decrease during stage I (the first 400 h), a slow decrease during stage II (from 400 to 3200 h), and the stable stage during stage III (up to 8800 h). P enrichment at the M23C6/γ interface is induced by the rejection of P during the formation of M23C6 carbide, and the M23C6/γ interface is preferred location for crack initiation. An increase in the amount of grain boundary M23C6 carbides increases the crack initiation sites, resulting in a rapid decrease of impact toughness during stage I. The ongoing rejection of Ni and Si during the coarsening of M23C6 carbide is beneficial for the nucleation of M6C carbide. The micro-crack initiation at the M23C6/M6C and M6C/γ interfaces is introduced, and a semi-continuous network of M23C6 carbide along grain boundary promotes the crack propagation, leading to a further decrease of impact toughness during stage II. The nearly continuous precipitate morphology is induced through continued growth of grain boundary M23C6 and M6C carbides. The propagation of micro-cracks along the M6C/γ, M23C6/γ, and M23C6/M6C interfaces leads to the mostly intergranular fracture, thus the impact energy tends to be stable during stage III.