There has been a wealth of recent work deriving optimal monetary policy utilising New Neo-Classical Synthesis (NNCS) models based on nominal inertia. Such models typically abstract from the impact of monetary policy on the government's finances, by assuming that consumers are infinitely-lived and taxes are lump-sum such that Ricardian Equivalence holds. In this paper, in the context of a sticky-price NNCS model, we assume that the government must adjust spending and/or distortionary taxation to satisfy its intertemporal budget constraint. We then consider optimal monetary and fiscal policies under discretion and commitment in the face of technology, preference and cost-push shocks. We find that the optimal precommitment policy implies a random walk in the steady-state level of debt, generalising earlier results that involved only a single fiscal instrument. In the case of negative fiscal shocks this implies permanently higher taxation and lower output and government spending to support the new steady-state debt shock, but the optimal combination of these variables will ensure a zero rate of inflation under commitment. We also find that the time-inconsistency in the optimal precommitment policy is such that governemnts are tempted, given inflationary expectations, to raise taxation to reduce the ultimate debt burden they need to service. Since taxation is a distortionary labour income tax, this aggresive raising of taxation raises firms' marginal costs and fuels inflation. We show that this temptation is only eliminated if following shocks, the new steady-state debt is equal to the original, first-best, debt level. This implies that under discretionary policy the random walk result is over-turned: debt will always be returned to this initial steady-state even although there is no explicit debt target in the government's objective function. In a series of numerical simulations we show that the welfare consequences of introducing debt are negligible for precommitment policies, but can be significant for discretionary policy.