AbstractTransmission electron microscopy (TEM) as well as X‐ray topography (XRT) and X‐ray diffractometry have been used for investigation of the structure of the LPE heteroepitaxial system In0.05Ga0.95As‐InyGa1−yAs1−xPx‐GaAs(111) A. A critical value of the lattice misfit has been shown to exist at the metallurgical boundary ((Δa/a)* ≈ 10−3) which results in the change of the film nucleation and growth mechanism as well as the change of misfit dislocations (MDs) generation mechanism. With (Δa/a)0 > (Δa/a)* the nucleation and growth mechanism is mixed: island growth at the first stages of growth and layer‐by‐layer growth at large thicknesses. MDs are created in an “island film” developing a non‐ordered dislocation network. The density of threading dislocations (Nd) is ∼ 108 cm−2. With (Δa/a)0 < (Δa/a)* there is layer‐by‐layer mechanism of film's nucleation and growth from the very first stages of crystallization. MDs are injected into continuous layer along the inclined slip planes {111}, thus forming a regular three‐dimensional grid of MDs. Nd is less than 106 cm−2 in the case. A model of dislocation structure formation in heterolayers has been proposed. Within the frame of this model the two critical values of phosphorus concentration in the quaternary melt have been quantitatively determined. These are corresponding to the change of MD generation mechanism. The expected values of Nd for (Δa/a)0 > (Δa/a)* and (Δa/a)0 < (Δa/a)* have been theoretically determined.