Building façade color plays a key role in shaping urban image, enhancing urban vitality, and optimizing citizens’ living experience. Moreover, colors can influence people’s perception of space, but the multiple interrelationships between color elements and users’ color evaluation and visual perception have not yet been thoroughly studied. In order to explore the relationships between color elements and visual perception and subjective comfort, this study discusses the matching relationship between color and the comfort of a residential building façade from three levels of color hue, lightness, and saturation by combining eye movement technology and a subjective evaluation. Taking Anshan San Village in Shanghai as the research object, the relationship model between color elements, visual perception, and subjective comfort is established through correlation analysis, linear regression, and multifactor regression analysis, and the correlation between single elements and multiple elements is explored from multi-dimensional perspectives. It aims to assess the operability of eye movement data as a measure of the visual comfort with the residential building façade color through multiple indicators and to explore and establish a new correlation assessment framework between the three attributes of façade color and users’ visual comfort. The results show that (1) highly saturated façade colors significantly increase cognitive load and decrease color evaluation. (2) Brightness has a non-linear relationship with visual perception and subjective evaluation, either too high or too low a brightness value will cause visual fatigue and thus reduce the color evaluation, and when the brightness is maintained at 40–80%, there will be a better color experience. (3) Warm tones such as red, orange, and yellow can strengthen the color perception to enhance the color evaluation of the façade; among cold tones, cyan and purple can also strengthen the visual perception of the color of the façade to enhance the color evaluation. After validation, the proposed new color comfort assessment method is feasible and comprehensive, and we hope it can provide a reference for the planning of old residential buildings and communities in the future.