This paper evaluates optimal public investment and fiscal policy for countries characterized by limited tax and debt capacities. We study a non stochastic CRS endogenous growth model where public expenditure is an input in the production process, in countries where distortions and limited enforceability result in limited fiscal capacities, as captured by a maximal effective tax rate. We show how persistent differences in growth rates across countries could stem from differential public finance constraints, and differentiate between the case where the public expenditure finances the flow of recurring spending (such as law enforcement), versus the stock of tangible public infrastructure. Although the flow of public expenditure raises productivity, the government should not borrow to finance it as the resulting increase in public debt would lower welfare and the growth rate. With outstanding public debt, the optimal fiscal policy should keep the debt-to-GDP ratio constant in the economy with or without a binding constraint on tax revenues as a share of GDP - current non-durable public goods should be financed only from current revenue. With investment in the stock of public infrastructure, public sector borrowing to finance the accumulation of public capital goods may allow the economy to reach a long-run optimal growth path faster. With a binding tax capacity constraint, if the ratio of the initial public/private sector stock of capital is smaller than the sustainable balanced growth ratio, the optimal policy for the government is to purchase public capital, financed by debt, to immediately attain the sustainable ratio of public capital to private capital. The sustainable steady-state ratio is endogenous to the initial public-to-private capital ratio, the tax capacity and any exogenous debt limit (say, due to sovereign risk). With capital stock adjustment costs, these statements apply to a transition of finite duration rather than an instantaneous stock jump. With either a binding exogenous debt limit or solvency constrained borrowing, a more patient country will have a higher steady-state growth rate but a lower steady-state public-to-private capital ratio.