Diamond detectors are known to suffer from the unwanted polarization phenomenon, when long-lived charge traps in the crystal bulk are charged during irradiation process and build up an electric field opposite to the bias field, deteriorating the detector performance. Here, we studied the polarization processes for diamond detectors made up of commercial high-quality single-crystal CVD diamonds in electric field of 1 V/μm with a focus on effect of light illumination on the polarization/depolarization dynamics, using a set of light emitting diodes operated from ultraviolet to red spectral range. The crystals irradiated with α-particles were characterized with the measurement of energy spectra and the transient current technique. The exposure to near-UV and blue light is found to induce a transient photocurrent (presumably via photoionization of nitrogen-related defects), which fills the charge traps, so the biased detector is polarized even if not irradiated, while longer wavelengths have a little effect. Without the biasing, the illumination speeds up the depolarization process, particularly, the exposure to 365 nm wavelength light can return the strongly polarized diamond detector into initial state in less than in 300 ms. The light illumination is concluded to be a convenient and speedy way to control/switch the polarization/depolarization processes of the diamond detectors.