The acceleration of urbanization has resulted in a heightened awareness of the impacts of urban environments on residents’ emotional states. This present study focuses on the Lixia District of Jinan City. By using urban street view big data and deep learning methods, we undertook a detailed analysis of the impacts of urban color features on residents’ emotional perceptions. In particular, a substantial corpus of street scene image data was extracted and processed. This was performed using a deep convolutional neural network (DCNN) and semantic segmentation technology (PSPNet), which enabled the simulation and prediction of the subjective perception of the urban environment by humans. Furthermore, the color complexity and coordination in the street scene were quantified and combined with residents’ emotional feedback to carry out a multi-dimensional analysis. The findings revealed that color complexity and coordination were significant elements influencing residents’ emotional perceptions. A high color complexity is visually appealing, but can lead to fatigue, discomfort, and boredom; a moderate complexity stimulates vitality and pleasure; high levels of regional harmony and aesthetics can increase perceptions of beauty and security; and low levels of coordination can increase feelings of depression. The environmental characteristics of different areas and differences in the daily activities of residents resulted in regional differences regarding the impacts of color features on emotional perception. This study corroborates the assertion that environmental color coordination has the capacity to enhance residents’ emotions, thereby providing an important reference point for urban planning. Planning should be based on the functional characteristics of the region, and color complexity and coordination should be reasonably regulated to optimize the emotional experiences of residents. Differentiated color management enhances urban aesthetics, livability, and residents’ happiness and promotes sustainable development. In the future, the influences of color and environmental factors on emotions can be explored in depth, with a view to assist in the formulation of fine urban design.