In this paper, we investigated the influence of the GaN buffer layer on the phonon escape time of phonon-cooled hot electron bolometers (HEBs) based on NbN material and compared our findings to conventionally employed Si substrate. The presented experimental setup and operation of the HEB close to the critical temperature of the NbN film allowed for the extraction of phonon escape time in a simplified manner. Two independent experiments were performed at GARD/Chalmers and MSPU on a similar experimental setup at frequencies of approximately 180 and 140 GHz, respectively, and have shown reproducible and consistent results. By fitting the normalized IF measurement data to the heat balance equations, the escape time as a fitting parameter has been deduced and amounts to 45 ps for the HEB based on Si substrate as in contrast to a significantly reduced escape time of 18 ps for the HEB utilizing the GaN buffer layer under the assumption that no additional electron diffusion has taken place. This study indicates a high phonon transmissivity of the NbN-to-GaN interface and a prospective increase of IF bandwidth for HEB made of NbN on GaN buffer layers, which is desirable for future THz HEB heterodyne receivers.