Ytterbium laser sources are state-of-the-art systems that are increasingly replacing Ti:Sapphire lasers in most applications requiring high repetition rate pulse trains. However, extending these laser sources to THz Time-Domain Spectroscopy (THz-TDS) poses several challenges not encountered in conventional, lower-power systems. These challenges include pump rejection, thermal lensing in nonlinear media, and pulse durations exceeding 100 fs, which consequently limit the detection bandwidth in TDS applications. In this article, we describe our design of a THz-TDS beamline that seeks to address these issues. We report on the effectiveness of temperature controlling the Gallium Phosphide (GaP) used to generate the THz radiation and its impact on increasing the generation efficiency and aiding pump rejection while avoiding thermal distortions of the residual pump laser beam. We detail our approach to pump rejection, which can be implemented with off-the-shelf products and minimal customization. Finally, we describe our solution based on a commercial optical parametric amplifier to obtain a temporally compressed probe pulse of 55 fs duration. Our study will prove useful to the increasing number of laboratories seeking to move from the high-energy, low-power THz time-domain spectroscopy systems based on Ti:Sapphire lasers, to medium-energy, high-power systems driven by Yb-doped lasers.