A microscopic theory is used to study the optical properties of semiconductor quantum dots. The dephasing of the coherent polarization due to carrier-carrier Coulomb interaction and carrier-phonon interaction is determined from quantum kinetic equations. We investigate the density dependence of the dephasing mechanisms, and compare the relevance of various interaction processes. The failure of frequently used approximations based on the GKBA with free single-particle energies is demonstrated for pure dephasing processes involving only the localized quantum-dot states.