By strongly reflecting solar radiation and being highly emissive within the atmospheric window, daytime radiative coolers can achieve sub-ambient temperature under direct sunlight. Radiative cooling performance is strongly coupled to specific climatic conditions since cooling efficiency is strongly affected by ambient air temperature, wind speed, and solar and ambient radiation intensity. In this paper, using a well-validated thermal model, the cooling performance of three radiative cooling materials with varying optical properties was evaluated under three distinct and representative climates. This analysis permits us to better understand the sensitivity of daytime radiative cooling materials to different climatic conditions, present strategies for selecting the ideal spectral properties of materials and investigate how to enhance cooling performance under adverse climatic conditions. It is shown that radiative cooling materials have better performance in hot and arid climates. Most radiative cooling materials exhibit the greatest response to changes in ambient radiation. Higher ambient air temperatures correspond to larger sub-ambient temperature of the surfaces, but this change is lower than that of the corresponding air temperature. Furthermore, by coupling a special optical grating window onto the surface of a radiative cooler, cooling performance can be significantly enhanced by asymmetrically reflecting incoming radiation but permitting outgoing emission. While an ideal material that only emits in the atmospheric window wavelengths presents the best performance under a large range of solar radiation, ambient radiation, and air temperature, the broadband ideal emitter exhibits higher cooling potential when coupled with the optical grating window.