Properties of the nematic–isotropic phase transition in polymer and low molecular weight liquid crystals doped with nanoparticles have been studied both experimentally and theoretically in terms of molecular mean-field theory. The variation of the transition temperature and the transition heat with the increasing volume fraction of CdSe quantum dot nanoparticles in copolymer and low molecular weight nematics has been investigated experimentally and the data are interpreted using the results of the molecular theory which accounts for a possibility of phase separation when the system undergoes the nematic–isotropic transition. The theory predicts that the nematic and isotropic phases with different concentrations of nanoparticles may coexist over a broad temperature range, but only if the nanoparticle volume fraction exceeds a certain threshold value which depends on the material parameters. Such unusual phase separation effects are determined by the strong interaction between nanoparticles and mesogenic groups and between nanoparticles themselves.