A non-empirical equation describing the effect of size on the temperature dependence of the optical bandgap of CdS (dEg/dT) is obtained on the basis of the Brus equation. Intriguingly, we find that dEg/dT diverges strongly from bulk values only within the "extreme confinement" (EC) regime. We conducted both experimental and theoretical investigations of the absorption spectra of CdS clusters and quantum dots as a function of temperature above room temperature. Our results show that the value of dEg/dT obtained from absorption spectra in the EC regime is 2.5 times higher than in the strong confinement regime. Notable ligand sensitivities are also observed for dEg/dT in the case of CdS clusters. Ab initio molecular dynamics simulations and density functional theory calculations reveal that thermal fluctuations are the crucial factor influencing the bandgap temperature coefficient. Our results help resolve some long-standing debates regarding the dEg/dT behavior of semiconductor quantum dots.