In some automatic systems, target detection is a common task, and visible images are common sources of raw data. Researchers have confirmed that polarization information highlights manmade targets. We propose an algorithm that fuses polarized and visible images to improve detection accuracy. First, the polarization parameter and visible images are simultaneously converted to the HSV color space. The initial fused image after adjusting the hue and saturation will be transformed into the lab color space. Then, the bisecting k-means algorithm is employed to segment the visible image. The visible and initial images are divided into three types of regions for color transfer in lab color space. Finally, the fused image is transformed back to the RGB color space, and the PolarLITIS data set is applied. The experimental results show that the gradient and contrast of the fused image are improved by 115% and 235.3%, respectively, compared with the visible image, and the final fused image is suitable to view with the naked eye. The proposed algorithm significantly improves accuracy.