We are interested in the phonon response in the frustrated magnets ${\text{SrCr}}_{9x}{\text{Ga}}_{12\ensuremath{-}9x}{\text{O}}_{19}$ (SCGO) and ${\text{Ba}}_{2}{\text{Sn}}_{2}{\text{ZnCr}}_{7x}{\text{Ga}}_{10\ensuremath{-}7x}{\text{O}}_{22}$ (BSZCGO). The motivation of the study is the recently discovered, phonon-driven, magnetic relaxation in the SCGO compound [Mutka et al., Phys. Rev. Lett. 97, 047203 (2006)] pointing out the importance of a low-energy $(\ensuremath{\hbar}\ensuremath{\omega}\ensuremath{\sim}7\text{ }\text{meV})$ phonon mode. In neutron-scattering experiments on these compounds, the phonon signal is partly masked by the magnetic signal from the Cr moments and we have therefore examined in detail the nonmagnetic isostructural counterparts ${\text{SrGa}}_{12}{\text{O}}_{19}$ (SGO) and ${\text{Ba}}_{2}{\text{Sn}}_{2}{\text{ZnGa}}_{10}{\text{O}}_{22}$ (BSZGO). Our ab initio lattice-dynamics calculations on SGO reveal a peak in the vibrational density of states matching with the neutron observations on SGO and SCGO. A strong contribution in the vibrational density of states comes from the partial contribution of the Ga atoms on the $2b$ and $12k$ sites, involving modes at the M point of the hexagonal system. These modes comprise dynamics of the Kagom\'e planes of the pyrochlore slab magnetic sublattice, $12k$ sites, and therefore can drive magnetic relaxation via spin-phonon coupling. Both BSZCGO and BSZGO show a similar low-energy Raman peak but no corresponding peak in the neutron-determined density of states of BSZGO is seen. However, a strong non-Debye enhancement of low-energy phonon response is observed. We attribute this particular feature to the Zn/Ga disorder on the $2d$ site, already evoked earlier to affect the magnetic properties of BSZCGO. We propose that this disorder-induced phonon response explains the absence of a characteristic energy scale and the much faster magnetic relaxation observed in BSZCGO.