Polarization imaging has become a promising way for clear underwater vision, which depends on the difference of polarization characteristics between backscattered light and target signal. In this paper, to achieve clear underwater color polarization imaging, we propose a learning-based method that uses multi-polarization fusion adversarial generative networks to learn the relationship between polarization information and object radiance. The proposed method is especially designed for multiple polarimetric images, which can effectively extract the polarization correlations of images with different polarization states. Moreover, to train the proposed network, we build, for the first time to our knowledge, a color polarization image dataset from natural underwater environments through passive polarization imaging. The experimental results in laboratory and natural underwater environments show that it is feasible to introduce polarization information into learning-based image recovery, and deep learning technology is conducive to the extraction of polarization information. Comparing with other methods, the proposed method can effectively remove the backscattered light and recover the object radiance.