We use deep surface photometry of the giant elliptical M49 (NGC 4472), obtained as part of our survey for diffuse light in the Virgo Cluster, to study the stellar populations in its outer halo. Our data trace M49's stellar halo out to ~ 100 kpc (7 Re), where we find that the shallow color gradient seen in the inner regions becomes dramatically steeper. The outer regions of the galaxy are quite blue (B-V ~ 0.7); if this is purely a metallicity effect, it argues for extremely metal poor stellar populations with [Fe/H] < -1. We also find that the extended accretion shells around M49 are distinctly redder than the galaxy's surrounding halo, suggesting that we are likely witnessing the buildup of both the stellar mass and metallicity in M49's outer halo due to late time accretion. While such growth of galaxy halos is predicted by models of hierarchical accretion, this growth is thought to be driven by more massive accretion events which have correspondingly higher mean metallicity than inferred for M49's halo. Thus the extremely metal-poor nature of M49's extended halo provides some tension against current models for elliptical galaxy formation.