Graphite components are constantly subjected to a combined actions of annealing and irradiation due to high temperatures and thermal spike caused by irradiation when reactors are operating, resulting in complex microstructures along with matching changes in the material properties and dimensions. This study investigates the evolution process of initial irradiation defects at different annealing temperatures, which is difficult to captured by experiment. The findings indicate that 923K reactor-temperature annealing can also quickly restore isolated self-interstitial atoms and small-sized point defect clusters to intralayer locations on the nanosecond scale. However, multi-interlayer penetration damages and localized point defect aggregation from high-energy irradiation can lead to the disappearance of layered structural information, which allows these damage structures to develop into interlayer dislocations during annealing process. These interlayer dislocations further exacerbate the interlayer expansion of graphite crystal and may elucidate the mechanism behind volume expansion in nuclear graphite within reactors. Fully amorphized regions can also regain a layered structure approximately when guided by residual layered structures at 2000K, while chaotic regions or disordered layered structures form in the absence of guidance. These chaotic regions significantly exacerbate the volume expansion of graphite model, which may be related to the rapid volume increase observed in nuclear graphite components at the end of reactor life. The study provides insights into the transformation of initial irradiation defects into different types of defects under different temperatures and damage states, which serve as a critical foundation for assessing the evolution of irradiation defects in nuclear graphite for reactor applications and annealing studies.