We show that it is possible to combine an early phase of String Gas Cosmology which can explain the origin of the observed structures on cosmological scales with a short later period of power law inflation which creates spatial flatness. The resulting model is consistent with the ``swampland criteria'' and the constraints coming from the {\it Trans-Planckian Censorship Conjecture}. Such a construction is not possible using only canonical slow-roll inflation, but it can emerge in the warm inflation scenario or in cold inflation with an exponential potential. The resulting cosmology is non-singular. We discuss the spectrum of cosmological perturbations resulting in our scenario. On large scales (scales which remain larger than the Hubble radius after the initial string gas phase) the spectrum is determined by the thermal string gas fluctuations set up in the primordial phase, and it is almost scale-invariant with a slight red tilt. On small scales, the perturbations produced during the inflationary phase dominate. On these scales, the spectrum is once again nearly scale-invariant. There is an intermediate range (scales which enter the Hubble radius during the period of inflation) where the string gas fluctuations are damped but continue to dominate over those produced during the period of inflation. On these scales the spectrum has a sharp red spectral index of $n_s - 1 \sim -2$.