The origination of magnetism in Co-doped ZnO-based dilute magnetic semiconductor (DMS) with room temperature ferromagnetism (RTFM) has some controversy. In this paper, Zn1-xCoxO nanoparticles (x = 0.00, 0.01, 0.03 and 0.05) were prepared using sol-gel method. The relationship between oxygen vacancy or carrier concentration and magnetic properties were investigated. The hexagonal wurtzite structure of all the samples was confirmed by X-ray diffraction (XRD) and Fourier-transform infrared spectroscopy (FTIR). The morphological features were investigated by scanning electron microscopy (SEM) and transmission electron microscope (TEM), and the optical properties, including energy gap, were analyzed by ultraviolet–visible spectroscopy (UV–vis). The number of oxygen vacancies (VO) was measured by X-ray photoelectron spectroscopy (XPS), which is positively correlated with the Co content. The RTFM properties were analyzed by physical properties measurement system (PPMS). The results reveal that a higher carrier concentration is critical to promoting ferromagnetism. And, the magnetism of Co-doped ZnO may originate from VO defects or the carrier.