The emotional health benefits of urban green space have been widely recognized. Flower borders, as a perennial plant landscape, have gradually become a current form of plant application in urban green spaces due to their rich color configurations. However, the related research primarily focuses on the impact of urban green spaces on public health, with relatively little attention given to how the colors of flower borders affect public emotional health. This study explored the relationship between the flower borders color characteristics and the public emotional health. In this study, 24 sample images were used as experimental materials, which selected based on their color richness and harmony. Additionally, face recognition technology and online random questionnaires were utilized to measure the public basic emotions and pleasure, respectively. The result shows that, based on the HSV color model and expert recommendations, 19 color characteristics were identified. The correlation analysis of the results from the public emotion with these color characteristics revealed that 13 color characteristics correlated with public emotional pleasure. Among them, blue, neutral purple, and low saturation were positively correlated. Through factor analysis, these thirteen color characteristics were summarized and categorized into four common factors (F1–F4), three of which are related to color. They are “low saturation of blue-violet percentage” (F1), “color configuration diversity” (F2), “bright red percentage” (F3), and “base green percentage” (F4), with F1 having the largest variance explained (27.88%). Finally, an evaluation model of color characteristics was constructed based on the variance explained by these four factors, which was demonstrated to effectively predict the level of public emotional pleasure when viewing flower borders. The results shed light on the effects of color characteristics on public emotions and provide new perspectives for subsequent flower border evaluations. Our results provide a valuable reference for future flower border color design, aiming to better improve public emotional health.