This work reports on the influence of the InP cap-layer growth rate on the structural and optical properties of InAs∕InP quantum dots (QDs) grown by metal-organic vapor phase epitaxy. A careful correlation between the structural and optical properties of the QDs completed by a modeling of their interband transition energy evidences the presence of different QD families with heights varying by monolayer steps. The analysis of transmission electron microscopy images and photoluminescence spectra demonstrates a drastic decrease of the QD height during the growth of the InP cap layer, due to As∕P exchange. The efficiency of this erosion mechanism is shown to be strongly related to the QD exposure time to PH3, depending on the growth rate of the InP cap layer.