When exposed to phosphorus, the (2×4) GaAs surface tends rapidly to roughen whereas it is not the case for the (2×4) InAs one, even after long exposure times. X-ray photoemission measurements show that the incorporated phosphorus amount is similar in both cases. The elastic energy difference between these two materials with incorporated phosphorus, calculated within the valence force field approximation, appears to be strong enough to explain this phenomenon. The role of surface reconstruction, mainly dimerization, on phosphorus incorporation is also emphasized.