We compute the graviton one-loop correction to the expectation value of the local expansion rate in slow-roll inflation, with both slow-roll parameters finite. The calculation is based on a recent method to explicitly construct gauge-invariant observables in perturbative quantum gravity at all orders in perturbation theory, and it is particularly suited in cases of highly-symmetrical space-time backgrounds. Our analysis adds to recent calculations of that correction in de Sitter space-time and in single-field inflation with constant deceleration. In the former case a vanishing one-loop correction was found, while in the latter the quantum backreaction produces a secular effect that accelerates the expansion. The quantum correction we describe here produces a finite secular effect that can either accelerated or decelerate the background expansion, depending on the value of the slow-roll parameters.