We studied systematically the fluorescence quenching of CdSe quantum dots (QDs) on graphene and its multilayers, as well as graphene oxide (GO) and reduced graphene oxide (rGO). Raman intensity of QDs was used as a quantitatively measurement of its concentration in order to achieve a reliable quenching factor (QF). It was found that the QF of graphene (∼13.1) and its multilayers is much larger than rGO (∼4.4), while GO (∼1.5) has the lowest quenching efficiency, which suggests that the graphitic structure is an important factor for quenching the fluorescence of QDs. It was also revealed that the QF of graphene is not strongly dependent on its thicknesses.