We present a comprehensive analysis of the $\gamma W$ interference radiative correction to the neutron $\beta$-decay matrix element. Within a dispersion relations approach, we compute the axial-vector part of the $\gamma W$ box amplitude $\Box^{\gamma W}_{A}$ in terms of the isoscalar part of the $F_3^{\gamma W}$ interference structure function. Using the latest available phenomenology for $F_3^{\gamma W}$ from the nucleon elastic, resonance, deep-inelastic, and Regge regions, we find the real part of the box correction to be $\Box^{\gamma W}_A = 3.90(9) \times 10^{-3}$. This improved correction gives a theoretical estimate of the CKM matrix element $|V_{ud}|^2=0.94805(26)$, which represents a 4$\sigma$ violation of unitarity.