We present late-time observations of the tidal disruption event candidate PS1-10jh. UV and optical imaging with HST/WFC3 localize the transient to be coincident with the host galaxy nucleus to an accuracy of 0.023 arcsec, corresponding to 66 pc. The UV flux in the F225W filter, measured 3.35 rest-frame years after the peak of the nuclear flare, is consistent with a decline that continues to follow a $t^{-5/3}$ power-law with no spectral evolution. Late epochs of optical spectroscopy obtained with MMT ~ 2 and 4 years after the peak, enable a clean subtraction of the host galaxy from the early spectra, revealing broad helium emission lines on top of a hot continuum, and placing stringent upper limits on the presence of hydrogen line emission. We do not measure Balmer H\delta absorption in the host galaxy strong enough to be indicative of a rare, post-starburst "E+A" galaxy as reported by Arcavi et al. (2014). The light curve of PS1-10jh over a baseline of 3.5 yr is best modeled by fallback accretion of a tidally disrupted star. Its strong broad helium emission relative to hydrogen (He II \lambda 4686/H\alpha > 5) could be indicative of either the hydrogen-poor chemical composition of the disrupted star, or certain conditions in the tidal debris of a solar-composition star in the presence of an optically-thick, extended reprocessing envelope.