We consider a class of exponential models of $f(R)$ gravity, in the presence of a canonical scalar field, for which at early times the effective Lagrangian of the theory becomes that of a rescaled canonical scalar field with the Einstein-Hilbert term becoming $\sim \alpha R$, with $\alpha$ a dimensionless constant. This rescaled Einstein-Hilbert scalar field theory at early times alters the inflationary phenomenology of well-known scalar field models of inflation, but more importantly, in the context of this rescaled theory, the Swampland criteria are easily satisfied, assuming that the scalar field is slowly rolling. We consider two models of inflation to exemplify our study, a fibre inflation model and a model that belongs to the general class of supergravity $\alpha$-attractor models. The inflationary phenomenology of the models is demonstrated to be viable, and for the same set of values of the free parameters of each model which ensure their inflationary viability, all the known Swampland criteria are satisfied too, and we need to note that we assumed that the first Swampland criterion is marginally satisfied by the scalar field, so $\phi\sim M_p$ during the inflationary era. Finally, we examine the late-time phenomenology of the fibre inflation potential in the presence of the full $f(R)$ gravity, and we demonstrate that the resulting model produces a viable dark energy era, which resembles the $\Lambda$-Cold-Dark-Mater model. Thus in the modified gravity model we present, the Universe is described by a rescaled Einstein-Hilbert gravity at early times, hence in some sense the modified gravity effect is minimal primordially, and the scalar field controls mainly the dynamics with a rescaled Ricci scalar gravity. However $f(R)$ gravity dominates at late-times, where it controls the dynamics, synergistically with the scalar field.