We show quantitatively how the limited polarization stability of vertical-cavity semiconductor lasers is related to the small dichroism and large quantum noise in these lasers. We introduce several techniques to measure this dichroism and noise. Suitable polarization projections are shown to yield either the heterodyne beat or the partition noise of the two polarization modes.