The origin of the $\mu$G magnetic fields observed in galaxies is unknown. One promising scenario is that magnetic fields generated during inflation, larger than 0.1 nG on Mpc scales, were adiabatically compressed to $\mu$G strengths in galaxies during structure formation. Thus, detecting a scale-invariant primordial magnetic field (PMF) above 0.1 nG on Mpc scales just after recombination would indicate an inflationary origin of galactic magnetic fields. This would also provide compelling evidence that inflation occurred since only an inflationary mechanism could generate such a strong, scale-invariant magnetic field on Mpc scales. In contrast, constraining the scale-invariant PMF strength to be below 0.1 nG would imply an inflationary scenario is not the primary origin, since such weak PMFs cannot be amplified enough via adiabatic compression to produce the strength of the galactic fields we observe today. We find that measurements of anisotropic birefringence by future CMB surveys will be able to improve the sensitivity to Mpc-scale inflationary PMFs by an order of magnitude, and, in particular, that CMB-HD would lower the upper bound to 0.072 nG at the 95% C.L., which is below the critical 0.1 nG threshold for ruling out a purely inflationary origin. If inflationary PMFs exist, we find that a CMB-HD survey would be able to detect them with about $3\sigma$ significance or higher, providing evidence for inflation itself.