Single crystal CVD (scCVD) diamond is an attractive material for particle detection in high energy physics for its good time resolution and reported outstanding radiation tolerance. In addition to direct signal loss via charge carrier trapping, polarization effect, caused by non-homogeneous filling of trap defects, is a known cause of signal degradation in irradiated scCVD diamond. This phenomenon was studied by intentionally polarizing irradiated diamonds. Even the relatively lightly irradiated (1014 protons/cm2) diamonds exhibited strong enough polarization to collapse the electric field with moderate rate of 5 MeV alpha particles. The transient current measurements were reproduced with TCAD simulations. The hypothesis that the polarization is caused by single neutral defect type in the bulk, was tested using two generic models. Neither one has a satisfactory agreement with the measurement data, which indicates that trapping at the interfaces play a significant role in space charge polarization.