We study the interaction of feedback from active galactic nuclei (AGN) and a multi-phase interstellar medium (ISM), in simulations including explicit stellar feedback, multi-phase cooling, accretion-disk winds, and Compton heating. We examine radii ~0.1-100 pc around a black hole (BH), where the accretion rate onto the BH is determined and where AGN-powered winds and radiation couple to the ISM. We conclude: (1) The BH accretion rate is determined by exchange of angular momentum between gas and stars in gravitational instabilities. This produces accretion rates ~0.03-1 Msun/yr, sufficient to power luminous AGN. (2) The gas disk in the galactic nucleus undergoes an initial burst of star formation followed by several Myrs where stellar feedback suppresses the star formation rate (SFR). (3) AGN winds injected at small radii with momentum fluxes ~L/c couple efficiently to the ISM and have dramatic effects on ISM properties within ~100 pc. AGN winds suppress the nuclear SFR by factors ~10-30 and BH accretion rate by factors ~3-30. They increase the outflow rate from the nucleus by factors ~10, consistent with observational evidence for galaxy-scale AGN-driven outflows. (4) With AGN feedback, the predicted column density distribution to the BH is consistent with observations. Absent AGN feedback, the BH is isotropically obscured and there are not enough optically-thin sightlines to explain Type-I AGN. A 'torus-like' geometry arises self-consistently as AGN feedback evacuates gas in polar regions.