The effect of irradiation with 3 MeV electrons and subsequent high-temperature annealing on the thermal conductivity κT of synthetic HPHT diamond containing impurity nitrogen at a concentration of about 100 ppm was studied at temperatures from 5 to 410 K. At each stage of the experiment, the optical absorption and photoluminescence spectra, magnetization and thermal conductivity were measured on two plates from the same diamond single crystal. Color centers and their concentration in different locations in the samples were determined from the optical spectra. The concentration of paramagnetic impurities in the samples was determined from the temperature and field dependencies of the paramagnetic magnetization. Thermal conductivity is suppressed by two orders of magnitude at temperatures from 35 to 50 K after irradiation to a fluence of 5×1018 cm−2. Annealing at a temperature of 1800° C restored the thermal conductivity of diamond at moderate and high temperatures, but not at temperatures below 120 K. The measured data on thermal conductivity were analyzed within the framework of the phenomenological Callaway model in order to understand the role of various phonon scattering processes and how it changes during irradiation and annealing.