We examined experimentally how the surface morphology, photoluminescence (PL) and X-ray diffraction depend on the amount of compressive strain in the InGaAsP layers grown by metal-organic vapor-phase epitaxy (MOVPE). It was found that there are two critical values of strain in the relaxation process. One is the threshold strain value ( ε c1) for the appearance of the cross-hatched pattern on the crystal surface and subsequent decrease in PL intensity. The other is that for the in-coherent growth ( ε c2), which leads to the broadening of the full-width half-maximum (FWHM) in the X-ray diffraction analysis. We also found that ε c1 is in good accordance with the force–balance theory, while ε c2 follows the energy–balance theory. We also examined their compositional wavelength ( λ g) dependence ( λ g of InGaAsP: 0.92 μm (InAsP), 1.15, 1.3 and 1.67 μm (InGaAs)). The ε c1 of InAsP is approximately half those of the other materials. This is probably due to the relatively large Poisson's ratio for InAsP. On the other hand, ε c2 of InGaAsP with λ g of 1.15 and 1.3 μm is slightly smaller than that of InAsP or InGaAs. This can be explained by local strain fluctuation in a nanoscale region, i.e. the fluctuation is stronger for quaternary InGaAsP compared to that for ternary InAsP or InGaAs. These findings are very useful for the crystal growth of high-quality highly strained InGaAsP.