Optical response of mixed-valent manganite $\mathrm{Tb}{\mathrm{Mn}}_{2}{\mathrm{O}}_{5}$ is studied in the spectral range from $0.6\phantom{\rule{0.3em}{0ex}}\text{to}\phantom{\rule{0.3em}{0ex}}5.8\phantom{\rule{0.3em}{0ex}}\mathrm{eV}$ in order to reveal the charge-transfer (CT) transitions responsible for the band-gap structure. Strong increase of optical response for the $z$ and $y$ polarizations and strong anisotropy with a relation ${ϵ}_{2x}<{ϵ}_{2y}<{ϵ}_{2z}$ are found. These findings agree with the theoretical analysis derived from the orientation of the ${\mathrm{Mn}}^{3+}{\mathrm{O}}_{5}$ pyramids in the unit cell and points to the one-center $p\text{\ensuremath{-}}d$ CT transitions as the main contributors to the spectral weight. We argue that the CT processes are accompanied by giant electric-dipole fluctuations and therefore may be a source of large electric polarization of ${\mathrm{Mn}}^{3+}{\mathrm{O}}_{5}$ pyramids due to the parity-breaking ${\mathrm{Mn}}^{3+}\text{\ensuremath{-}}{\mathrm{Mn}}^{4+}$ isotropic exchange interaction.