We propose and demonstrate a method to increase the momentum separation between the arms of an atom interferometer and thus its area and measurement precision, by using Bloch oscillations (BOs) in an excited band of a pulsed optical standing wave lattice. Using excited bands allows us to operate at particular "magic" depths, where high momentum transfer efficiency ($>99.4\%$ per $\hbar k$, where $\hbar k$ is the photon momentum) is maintained while minimizing the lattice-induced phase fluctuations ($<1\,$milliradian per $\hbar k$) that are unavoidable in ground-band BOs. We apply this method to demonstrate interferometry with up to 40$\hbar k$ momenta supplied by BOs. We discuss extensions of this technique to larger momentum transfer and adaptations towards metrological applications of atom interferometry such as a measurement of the fine-structure constant.