The lightness of an object is an intrinsic property that depends on its surface reflectance spectrum. The visual system estimates an object's lightness from the light reflected off its surface. However, the reflected light also depends on object extrinsic properties of the scene, such as the light source. For stable perception, the visual system needs to discount the variations due to the object extrinsic properties. We characterize this perceptual stability for variation in two spectral properties of the scene: the reflectance spectra of background objects and the intensity of light sources. We measure human observers' thresholds of discriminating computer-generated images of 3D scenes based on the lightness of a spherical target object in the scene. We measured change in discrimination thresholds as we varied the reflectance spectra of the objects and the intensity of the light sources in the scene, both individually and simultaneously. For small amounts of extrinsic variations, the discrimination thresholds remained nearly constant indicating that the thresholds were dominated by observers' intrinsic representation of lightness. As extrinsic variation increased, it started affecting observers' lightness judgment and the thresholds increased. We estimated that the effects of extrinsic variations were comparable to observers' intrinsic variation in the representation of object lightness. Moreover, for simultaneous variation of these spectral properties, the increase in threshold squared compared to the no-variation condition was a linear sum of the corresponding increase in threshold squared for the individual properties, indicating that the variations from these independent sources combine linearly.