The effects of color and lighting on thermal perception and comfort have been studied through a multitude of methods, including virtual reality (VR) simulations as well as in vivo studies. The potential role of surface materials, however, remains poorly understood in this regard, even though materials and surface colors are clearly related factors in the perception of building interiors. In this study, we tested two Virtual Reality (VR) office spaces, one designed with wood and one with concrete. 38 participants (N = 38) were recruited to complete a simple reading task over fifteen minutes while being seated in a controlled temperature thermal booth. During this time, the booth temperature dropped by an average of 0.3 °C (0.5 °F) per minute from 22.2 °C (72 °F) to 17.7 °C (64 °F). Participants had the perceived ability to control the room’s temperature via a virtual reality interface, an action that served as a measure of “thermal tolerance” based on how long it took participants to try and stop the constant temperature drop in the thermal booth during the experimental task. Participants were surveyed after the study to state their perceived starting and ending room temperature. The results confirmed the hypothesis that the wood room was perceived as warmer overall than the concrete room. However, the results for the “thermal tolerance” metric varied based on the current outside temperature as we predict participants expected different interior temperature regimes in warm compared to cool outdoor conditions. The perception of the concrete room as cooler than the wood room showed the greatest correllation with thermal tolerance when outdoor temperatures were warm, increasing thermal comfort in the concrete room. The results of the study validate the hue-heat hypothesis for architectural building materials and support the potential for thermal perception to induce actionable behavior and promote passive energy savings through the choice of interior building materials.