Using large (512-atom) pseudopotential supercell calculations, we have investigated the composition dependence of the momentum matrix element ${M}_{v,c}$ for transitions between the valence-band maximum and the conduction-band minimum of three semiconductor alloys: ${\mathrm{GaP}}_{1\ensuremath{-}x}{\mathrm{N}}_{x}$ and ${\mathrm{GaAs}}_{1\ensuremath{-}x}{\mathrm{N}}_{x},$ exhibiting large chemical and size differences between their alloyed elements, and ${\mathrm{GaP}}_{1\ensuremath{-}x}{\mathrm{As}}_{x},$ which is a weakly perturbed alloy. In the composition ranges where these alloys have a direct band gap, we find that (i) in ${\mathrm{GaP}}_{1\ensuremath{-}x}{\mathrm{As}}_{x},$ ${M}_{v,c}$ is large (like the virtual-crystal value) and nearly composition independent; (ii) in ${\mathrm{GaAs}}_{1\ensuremath{-}x}{\mathrm{N}}_{x},$ ${M}_{v,c}$ is strongly composition dependent: large for small $x$ and small for large $x;$ and (iii) in ${\mathrm{GaP}}_{1\ensuremath{-}x}{\mathrm{N}}_{x},$ ${M}_{v,c}$ is only slightly composition dependent and is significantly reduced relative to the virtual-crystal value. The different behavior of ${\mathrm{GaP}}_{1\ensuremath{-}x}{\mathrm{As}}_{x},$ ${\mathrm{GaP}}_{1\ensuremath{-}x}{\mathrm{N}}_{x},$ and ${\mathrm{GaAs}}_{1\ensuremath{-}x}{\mathrm{N}}_{x}$ is traced to the existence/absence of impurity levels at the dilute alloy limits: (a) there are no gap-level impurity states at the $x\ensuremath{\rightarrow}1$ or $x\ensuremath{\rightarrow}0$ limits of ${\mathrm{GaP}}_{1\ensuremath{-}x}{\mathrm{As}}_{x},$ (b) an isolated As impurity in GaN ($\mathrm{GaN}:\mathrm{A}\mathrm{s}$) has a deep band gap impurity level but no deep impurity state is found for N in GaAs, and (c) $\mathrm{GaN}:\mathrm{P}$ exhibits a P-localized deep band-gap impurity state and $\mathrm{GaP}:\mathrm{N}$ has an N-localized resonant state. The existence of deep levels leads to wave-function localization in real space, thus to a spectral spread in momentum space and to a reduction of ${M}_{v,c}.$ These impurity levels are facilitated by atomic relaxations, as evident by the fact that unrelaxed $\mathrm{GaN}:\mathrm{A}\mathrm{s}$ and $\mathrm{GaN}:\mathrm{P}$, show no deep levels, have extended wave functions, and have large interband transition elements.