Control of thermal transport in colloidal crystals plays an important role in modern technologies. A deeper understanding of the governing heat transport processes in various systems, such as polydisperse colloidal crystals, is required. This study shows how strongly the particle size polydispersity of a model colloidal crystal influences the thermal conductivity. The thermal conductivity of model colloidal crystals has been calculated using molecular dynamics simulations. The model crystals created by particles interacting through Yukawa (screened-Coulomb) interaction are assumed to have a face-centered cubic structure. The influence of the Debye screening length, contact potential, and particle size polydispersity on the thermal conductivity of Yukawa crystals was investigated. It was found that an increase in particle size polydispersity causes a strong—almost fivefold—decrease in the thermal conductivity of Yukawa crystals. In addition, the obtained results showed that the effect of the particle size polydispersity on reducing the thermal conductivity of Yukawa crystals is stronger than changes in values of the Debye screening length or the contact potential.