Behavioral state modulates neural activity throughout the visual system; this is largely due to changes in arousal that alter internal brain state. However, behaviors are constrained by the external environmental context, so it remains unclear if this context itself dictates the regime of visual processing, apart from ongoing changes in arousal. Here, we addressed this question in awake head-fixed mice while they passively viewed visual stimuli in two different environmental contexts: either a cylindrical tube, or a circular running wheel. We targeted high-density silicon probe recordings to the dorsal lateral geniculate nucleus (dLGN) and simultaneously measured several electrophysiological and behavioral correlates of arousal changes, and thus controlled for them across contexts. We found surprising differences in spatial and temporal processing in dLGN across contexts, even in identical states of alertness and stillness. The wheel context (versus tube) showed elevated baseline activity, faster visual responses, and smaller but less selective spatial receptive fields. Further, arousal caused similar changes to visual responsiveness across all conditions, but the environmental context mainly changed the overall set-point for this relationship. Together, our results reveal an unexpected influence of the physical environmental context on fundamental aspects of visual processing in the early visual system.