The contribution of one loop milli-charged fermion vacuum polarization in cosmic magnetic field to the cosmic microwave background (CMB) polarization is considered. Exact and perturbative solution of the density matrix equations of motion in terms of the Stokes parameters are presented. For linearly polarized CMB at decoupling time, it is shown that propagation of CMB photons in cosmic magnetic field would generate elliptic polarization (circular and linear) of the CMB due to milli-charged fermion vacuum polarization. Analytic expressions for the degree of circular polarization and rotation angle of polarization plane of the CMB are presented. Depending on the ratio of milli-charged fermion relative charge to mass, $\epsilon/m_\epsilon$, and CMB observation frequency, it is shown that the acquired CMB degree of circular polarization could be of the order of magnitude $P_C(T_0)\sim 10^{-10}- 10^{-6}$ in the best scenario. The effect studied generates CMB polarization even in the case when the CMB is initially in thermal equilibrium. Limits on the magnetic field amplitude due to prior decoupling CMB polarization are presented.