The low-temperature thermal conductivity ($\ensuremath{\kappa}$) of GdFe${O}_{3}$ single crystals is found to be strongly dependent on magnetic field. The low-field $\ensuremath{\kappa}(H)$ curves show two ``dips'' for $H\ensuremath{\parallel}a$ and only one dip for $H\ensuremath{\parallel}c$, with the characteristic fields having good correspondence with the spin-flop and the spin-polarization transitions. A remarkable phenomenon is that the sub-Kelvin thermal conductivity shows hysteretic behaviors on the history of applying the magnetic field; that is, the $\ensuremath{\kappa}(H)$ isotherms measured with increasing field are larger than those with decreasing field. Intriguingly, the broad region of magnetic field ($~$0--3 T) showing the irreversibility of heat transport coincides with that presenting the ferroelectricity. It is discussed that the irreversible $\ensuremath{\kappa}(H)$ behaviors are due to the phonon scattering by ferroelectric domain walls. This result shows an experimental feature that points to the capability of controlling the ferroelectric domain structures by magnetic field in multiferroic materials.