The annealing process of nuclear graphite accompanying the release of Wigner energy was studied in this research. IG-110U and ETP-10 graphite were irradiated at 200 °C by neutrons with low-dose (1.38 × 1019 n/cm2) and high-dose (1.92 × 1020 n/cm2), respectively. Their Wigner energy release at 100–550 °C was measured based on a high-precision differential scanning calorimeter. Characteristics of Wigner energy release spectrum were investigated. The activation energy of the energy release peak (170–230 °C) for low-dose graphite was 1.48 eV, which was related to the decomposition of interstitial clusters and the behaviour of subsequent interstitial pairs. Furthermore, a comprehensive analysis of Wigner energy release combined with thermal conductivity and multi-scale structural parameters during the neutron irradiation and annealing process is given.