Diamond synthesis with silicon-vacancy (SiV) color centers is of high interest to nanophotonics and optical quantum technologies. The methods to control and/or maximize concentration of Si atoms incorporated in diamond lattice, and coupled to vacancies, together with improving crystallinity of diamond structure, are in demand. Here, we studied the effect of heat treatment of electron-irradiated single crystal CVD diamond doped with 28S isotope, on evolution of the neutral SiV0 and negatively charged SiV− centers. The crystal subjected to stepwise annealing at temperatures Tann from 200 °C to 1640 °C has been analyzed with photoluminescence (PL) and optical absorption spectroscopies at room and low-temperature (T = 5 K). We found a complex non-monotonous behavior of SiV0 and SiV− intensity both in absorption and PL, with sharp rise at Tann > 600 °C, reaching a maximum concentration peak, decline at 850 °C, and a repeated elevation to higher Tann, reflecting creation and annihilation processes of these defects. Moreover, we detected a group of seven lines of a Si-related defect in the range 828–871 nm, which strongly correlate with annealing dynamics of the SiV− and, especially, of SiV0 centers, and estimated the isotope spectral shift. In parallel, dynamics of other optical centers such as NV, R11 and GR1, in course of the annealing is traced. The controlled annealing opens the way to preparation of the SiV color centers with optimized optical properties, promising for quantum technologies.