Polarons can control carrier mobility and can also be used in the design of quantum devices. Although much effort has been directed into investigating the nature of polarons, observation of defect-related polarons is challenging due to electron-defect scattering. Here we explore the polaronic behavior of nitrogen-vacancy (NV) centers in a diamond crystal using an ultrafast pump-probe technique. A 10-fs optical pulse acts as a source of high electric field exceeding the dielectric breakdown threshold, in turn exerting a force on the NV charge distribution and polar optical phonons. The electronic and phononic responses are enhanced by an order of magnitude for a low density of NV centers, which we attribute to a combination of cooperative polaronic effects and scattering by defects. First-principles calculations support the presence of dipolar Fröhlich interaction via non-zero Born effective charges. Our findings provide insights into the physics of color centers in diamonds.