Single quantum-dot spectroscopy has revealed characteristic but so far unexplained differences in the optical spectra from different quantum-dot systems. We propose a size-dependent accumulation of carriers as the dominant mechanism behind these differences. We support our hypothesis with photoluminescence spectroscopy on single InP/GaInP quantum dots positioned below a transparent Schottky gate. We show that without external bias, the dots are filled with 15--20 electrons. By applying a reverse bias, we are able to reduce the electron accumulation while monitoring the evolution of the emission spectrum. We find that the emission peaks disappear one by one until, at a sufficiently low number of electrons in the dot, the remaining broad peaks are replaced by numerous very sharp peaks.