In the field of art, color matching is widely used in various art designs, such as images, posters, clothing, and interior home design. Among them, harmonious color matching is the decisive factor in whether a design is popular or not. To solve the problem of estimating color harmony, this study analyzes from the perspective of color pairs and uses the two-layer maximum likelihood estimation method to make preliminary predictions of color harmony by statistically modeling paired color preferences in existing datasets. After obtaining the preliminary estimation of color harmony, multiple linear regression is selected for denoising processing. Subsequently, the preliminary prediction results were refined using a backpropagation neural network, extracting various color features in different color spaces, and ultimately obtaining accurate harmony estimates. The results indicate that, compared with existing methods, the proposed method can simulate the aesthetic cognition of different users towards different color themes. Under the same statistical method, the model can maintain good harmony estimation and experimental results. This method can promote the development of related research fields, such as quickly evaluating the color harmony of an image, and one click color changing in scenes such as clothing, home, 3D models, etc. according to different user needs.