We study the thermal conductivity of single-walled carbon nanotube bundles and multi-walled carbon nanotubes. It is shown that, for an individual single-walled carbon nanotube, its thermal conductivity is both diameter and chirality dependent. In defect-free bundles and multi-walled carbon nanotubes, the phonon essentially does not transcend the bonds between the constituent walls due to the weak intertube interaction. If, however, the intertube coupling is strong, a substantial reduction in the thermal conductivity maybe implied. Such a low thermal conductivity can be found in several thermal transport experiments of carbon nanotube mats.