An evaluation for objectively assessing the quality of visible and infrared color fusion image is proposed. On the basis of the consideration that human perception is most sensitive to color, sharpness, and contrast when assessing the quality of color image, we propose four objective metrics: image sharpness metric (ISM), image contrast metric (ICM), color colorfulness metric (CCM), and color naturalness metric (CNM). The ISM is evaluated by image gradient information. The ICM is defined based on both gray and color histogram characteristics. A color chroma metric, as well as a color variety metric based on a color difference gradient, is proposed, respectively, to define the CCM. The CNM is defined by measuring the color distribution's similarity between the fusion image and nature image, which are of the same scene. All the color attributions are computed in the CIELAB color space. Experimental results show that the proposed objective metrics are meaningful and effective on color fusion image evaluation because they correspond well to subjective evaluation.