Specular highlights on the surface of an object can serve as cues for estimating illumination color, because highlights on many objects directly reflect illumination colors. In addition, we experience that illumination colors typically have chromaticities near the daylight locus in daily life, and this experience may possibly affect our estimation of the illumination color. In this study, we investigated the effects of specular highlights and object colors on the estimation of the illumination colors by human observers using images of a single object with specular highlights as stimuli. In the experiments, the colors of all pixels on an object with specular highlights changed from the reference D65 chromaticity to a test chromaticity in a trial. Observers judged whether this chromaticity change appeared to be induced by changing the object or by changing the illumination. In the control experiment, observers performed the same task for a stimulus of a mat object without specular highlights. Different values of hue and saturation were used as the test chromaticities. In the results, the chromaticity changes appeared more likely to be caused by changing the illumination when the test color had lower saturation. In addition, the ratios of “illumination changed” responses were slightly higher for objects with specular highlights than for objects without specular highlight. However, the ratios of “illumination changed” responses averaged across observers were not largely different across color directions. These results suggest that specular highlights slightly contribute to the estimation of illumination color, and that the daylight locus does not largely affect the estimation of the illumination color.