We compare the dynamical masses of dwarf galaxies in the Local Group (LG) to the predicted masses of halos in the ELVIS suite of $\Lambda$CDM simulations, a sample of 48 Galaxy-size hosts, 24 of which are in paired configuration similar to the LG. We enumerate unaccounted-for dense halos ($V_\mathrm{max} \gtrsim 25$ km s$^{-1}$) in these volumes that at some point in their histories were massive enough to have formed stars in the presence of an ionizing background ($V_\mathrm{peak} > 30$ km s$^{-1}$). Within 300 kpc of the Milky Way, the number of unaccounted-for massive halos ranges from 2 - 25 over our full sample. Moreover, this "too big to fail" count grows as we extend our comparison to the outer regions of the Local Group: within 1.2 Mpc of either giant we find that there are 12-40 unaccounted-for massive halos. This count excludes volumes within 300 kpc of both the MW and M31, and thus should be largely unaffected by any baryonically-induced environmental processes. According to abundance matching -- specifically abundance matching that reproduces the Local Group stellar mass function -- all of these missing massive systems should have been quite bright, with $M_\star > 10^6M_\odot$. Finally, we use the predicted density structure of outer LG dark matter halos together with observed dwarf galaxy masses to derive an $M_\star-V_\mathrm{max}$ relation for LG galaxies that are outside the virial regions of either giant. We find that there is no obvious trend in the relation over three orders of magnitude in stellar mass (a "common mass" relation), from $M_\star \sim 10^8 - 10^5 M_\odot$, in drastic conflict with the tight relation expected for halos that are unaffected by reionization. Solutions to the too big to fail problem that rely on ram pressure stripping, tidal effects, or statistical flukes appear less likely in the face of these results.